Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs
Authors: Jie Zhang
Zhongqi Wang
Mengqi Lei
Zheng Yuan
Bei Yan
Shiguang Shan
Xilin Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmark
Large Vision language Model
Perception Ability
Primary area: datasets and benchmarks

Abstract: Currently many benchmarks have been proposed to evaluate the perception ability of the Large Vision-Language Models (LVLMs).
However, most benchmarks conduct questions by selecting images from existing datasets, resulting in the potential data leakage. Besides, these benchmarks merely focus on evaluating LVLMs on the realistic style images and clean scenarios, leaving the multi-stylized images and noisy scenarios unexplored. In response to these challenges, we propose a dynamic and scalable benchmark named Dysca for evaluating LVLMs by leveraging synthesis images. Specifically, we leverage Stable Diffusion and design a rule-based method to dynamically generate novel images, questions and the corresponding answers. We consider 51 kinds of image styles and evaluate the perception capability in 20 subtasks. Moreover, we conduct evaluations under 4 scenarios (i.e., Clean, Corruption, Print Attacking and Adversarial Attacking) and 3 question types (i.e., Multi-choices, True-or-false and Free-form). Thanks to the generative paradigm, Dysca serves as a scalable benchmark for easily adding new subtasks and scenarios. A total of 24 advanced open-source LVLMs and 2 close-source LVLMs are evaluated on Dysca, revealing the drawbacks of current LVLMs. The benchmark is released in anonymous github page \url{https://github.com/Benchmark-Dysca/Dysca}.

Published as a conference paper at ICLR 2025
DYSCA: A DYNAMIC AND SCALABLE BENCHMARK FOR EVALUATING PERCEPTION ABILITY OF LVLMS
Jie Zhang'?, Zhongqi Wang Mengqi Lei?, Zheng Yuan 1
Bei , Shiguang S2 Xilin 12
1 Key Laboratory of AI Safety of CAS, Institute of Computing Technology.
Chinese Academy of Sciences (CAS), Beijing, China
2 University of Chinese Academy of Sciences, Beijing, China
3 China University of Geosciences
ABSTRACT
Currently many benchmarks have been proposed to evaluate the perception ability of the Large Vision-Language Models (LVLMs). However, most benchmarks conduct questions by selecting images from existing datasets, resulting in the potential data leakage. Besides, these benchmarks merely focus on evaluating LVLMs on the realistic style images and clean scenarios, leaving the multi-stylized images and noisy scenarios unexplored. In response to these challenges, we propose a dynamic and scalable benchmark named Dysca for evaluating LVLMs by leveraging synthesis images. Specifically, we leverage Stable Diffusion and design a rule-based method to dynamically generate novel images. questions and the corresponding answers. We consider 51 kinds of image styles and evaluate the perception capability in 20 subtasks. Moreover, conduct evaluations under 4 scenarios (i.e., Clean, Corruption, Print Attacking and Adversarial Attacking) and 3 question types (i.e., Multi-choices, True-or-false and Free-form). Thanks to the generative paradigm, Dysca serves as scalable benchmark for easily adding new subtasks and scenarios. total of 24 advanced open-source LVLMs and 2 close- source LVLMs are evaluated on Dysca, revealing the drawbacks of current LVLMs. The benchmark is released at / htt.t/ttttuuuu/.// com Robi n WWC/Dyscs
INTRODUCTION
Recent years have witnessed the great success of the Large Vision-Language Models (LVLMs) LLi
ct al. 2023d Zhu et al.] 2023 Dai et al. 2023 Liu et al. 20236 Li et al. 2023a Chen al.
2023b Zhang et al.) 2023 Su et al. 2023 Gong et al. 2023 Sun et al. 2023b). These models
leverage the powerful Large Language Models (LLMs) CChung el al. 2022 OpenAT 2022] Touvron
ct al. 2023 OpenAl 2023 FastChat 2023) as their brain and incorporate the state-of the-the-ttt visual encoders (Radford et al. [021]] Fang et al. 2023; Dosovitskiy et al. 2020) as their eyes. Thanks to
the alignment of visual feature with textual space and the development of visual instruction tuning techniques (Liu et al. 20235). LVLMMs showcase the impressive capability in terms of visual scene comprehension and multimodal instruction-followingg
In order to comprehensively evaluate the capabilities of LVLMs, many benchmarks have been purposed (Antol et al.) 2015a Singh al., 2019 Xu et al. 2023 Shao al. 2023
Li et al. 2023c b
Fu et al. 2023) Bai et al. (223b) Yu et al. 2023 Yang et al.
2023b Chen et al.
2024). where we
categorize the current benchmarks into three types (Fu al. 2023). The first type the classical
benchmarks, such as COCO Caption (Chen et al. 2015) and VQA (Antol et al. [2115]] Goyal et al.
2017 Marino et al.) 2019). Although these benchmarks provide high-quality evaluation data, they also have notable limitations. On the one hand, they are inadequate for measuring the fine-grained capabilities of current LVLMs, offering the limited insightful feedback for the future improvement. On the other hand, since these classical benchmarks have been available the open-source test data for a long time, it is hard to prevent the data leakage problem. The second type of benchmarks evaluate the LVLMs through subjective manner (Yang et al.) 2023b Wu al. 2023). Although the benchmarks reveal the insightful drawbacks of current models, their data scale is limited (i.e., less than 200 annotations) and they require manual evaluation by experts. The third type is built

---

Published as a conference paper at ICLR 2025
(a) VQA Generation
Style
Attribute
Foreground
Background
Generated
Prompis
Scenario
Type
Walt axredl
Turkk Tigi Can Moso
Reperceliitic
Blue Coke:
Flnd
Indocr
whie ooker bepernalinic etyle sront-commoo ground
Cless
Muttipie Choices
Wasrcoker
White Color
Tixi
Sapw-cphered
Lise Skazch
Print Armakk
immgeu False
a paple coloe iefiiacrator
Manga
you
Objoc:
Erbon Street
True Filse
Storyy
Deflusion Modid
Advenarial Artrrk
Faiy Tale
25 your
Harran
Sorry Sky
Smiling
Sooe smiling, Mins
Compriio
Image Cagtioring
(A)
Fincl Ant
Shorting
Landrart
Mars
Q: reserbeerbe fiis eage in somn ofooneee eyfo arrituas danak.
(b) VQA Cleaning
(c) Evaluating LVLMs
Automatically Cleaning
Filtering low quilty VQAx
VQA gencrated in Step (a)
Dysca
LVLMs
Evaluation Results
Figure I: Overview of the automatic pipeline for generating Vision-language QAs, cleaning Vision- language QAs and evaluating LVLMs. (a) We first constructs prompts in terms of content, style and background, leveraging the text-to-Image (T2I) diffusion model (e.g.. SDXL (Podell et al. (023)) to synthesis images to be asked. Then based the scenarios and the question type, post-process the synthesis images and generate the specific textual questions, respectively. (b) We further filter out low quality Vision-language QAs by utilizing trained models to form the final Dysca. (c) Finally, evaluate LVLMs on our Dysca and feedback the fine-grained evaluation results.
for objectively evaluating current LVLMs and the comparison between them are shown in Tab. [ They provide an objective and automatic evaluation manner, giving the fine- ene-grained evaluation for the LVLMs. However, these benchmarks conduct Vision-language QAs by selecting images from existing dataset and annotate the textual questions. Although they claim that the questions are re-annotated, the previous work (Chen et al. 2024) has demonstrated that these benchmarks have Models unintentionally leaked into the training data of LLMs and LVLMs. Besides, most benchmarks focus on evaluating LVLMs in the realistic images and clean scenarios, leaving the multi stylized images and noisy scenarios unexplored. While some works like MMCBench (Zhang et al. 2024b) and Typographic Dataset (Cheng et al.] 2024) have investigated the robustness of LVL.Ms with corrupted and print-attacked images, respectively, they have not explored the effect of these noisy images on various perceptual tasks.
In this paper, aiming to address these challenges above, propose Dysca which is dynamic and scalable benchmark for evaluating the perception ability of LVLMs via various subtasks and scenarios. Inspired by the prior evaluation works for LLMs (Liang et al. 2023), we investigate whether we could leverage the large-scale synthesized images for evaluating LVLMs. We display the overview of our pipeline in Fig. [ Specifically, we leverage Stable Diffusion and design rule-based method to dynamically generate novel images, questions and corresponding answers. We decouple the prompt into part, i.c., attribute, foreground, style and background, and design pre-defined templates to dynamically generate prompts, as displayed in Fig. 3 Then we utilize the state-of-the-art text-to-image diffusion models (e.g., SDXL (Podell et al. (023)) to generate the corresponding images. Since we already know the main information of the images through prompts, we easily generate question- answer textual pairs by the rule-based method. After that, in order to obtain the high quality Vision- QAs, we employ CLIP (Radford et al. (021) to perform data cleaning on the generated Vision-language QA pairs. Dysea focuses on assessing the fine- fine-grained perception abilities, including recognizing human, animal, object, landmark, etc. Dysca evaluates LVLMs with 20 perceptual subtasks, containing a total number of 51 different artistic styles. Besides, evaluate the robustness of the models across different scenarios and question types, construct 4 testing scenarios (clean, corruption, print attacking and adversarial attacking) and question types (multi-choices, true-or-false and free-form questions).
Compared to previous works Tab. I we provide an end-to-end process from image to Vision-QU QA generation. This approach significantly reduces annotation costs compared manually labeling
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons between existing LVLM benchmarks. X indicates that the benchmarks include both newly collected images / annotationss and images / annotations gathered from existing datasets. The scale of released benchmark is 617K, however Dysca is able to generate unlimited data to be tested.
#Evaluation Data Scale
#Perceptual Tasks
Automatic Annotation
Novel Images Novel Questions
Question Type
Automatic
Benchmark
Evaluation
LLaVA-Beneh Bench
0.15K
Free-form
MME
2.3K
10
X
Truc-or-falsc
LVLM-cHub
Free-form
tiny-LVLM-eHall
2.1K
3
X
Frree-fom
SLED-Bench
19K
X
Multi-choices
MMBench
2.9K
12
X
Multi-choices
TouchStone
0.9K
10
X
Free-form
REFORM-EVAL
50K
X
Multi-choices
MencigBench
30K
6
X
Multi-choices
MM-VET
0.2K
4
X
K
Free-form
MLLM-Bennch
0.42K
K
Free-form
SEED-Bench2
24K
10
X
Multi-choices
BenchL
2.4K
X
Free-form
Free-form
JourneyDB
5.4K
Multi-choices
Free-form
Dysca (Ours)
617K
20
Multi-choices
Truc-or-false
images (e.g., MME (Fu et al. 2023)) while achieving the correctness for evaluating LVLMs.
also avoids the risk of hallucinate annotations that may occur when using ChatGPT for labeling based on image prompts (e.g.. JourneDBB (Sun et al. 2023a)). This novel pipeline enables us to create benchmark that is easily scalable and adaptable for incorporating new subtasks and scenarios. Thanks to the generative paradigm, Dysca can be customized to meet the specific requirements of the evaluator for testing purposes.
In summary, our work makes the following key contributions:
Dynamic and Scalable Benchmark: We propose Dysca, benchmark that is able to dynamically generate the test data that users need and is easily to scale up to to new subtasks and scenarios.
Multi-grained Perceptual Subtasks and Multi-scenarios: Dysca aims testing LVLMs' performance on diverse styles, 4 image scenarios (i.e., clean, corruption, print attacking and adversarial attacking) and 3 question types (i.e., multi-choices, true-or-false and free-form questions), reporting the 20 perceptual subtasks performance of 26 mainstream LVLMs, including GPT-4o (Ope) and Gemini-L.5-Prre TTam et 2024).
Analysis and Observations: We demonstrate for the first time that evaluating LVLMs using large-scale synthetic data is valid. Experiments show the strong correlation coefficient between our evaluation rankings and the rankings obtained from non-synthetic benchmarks. The evaluation results also reveal the weakness of current LVLMs when facing different question types, image styles and image scenarios.
RELATED WORKS
2.1
LARGE VISION-LANGUAGE MODELS
The landscape of Large Vision-Languaae Models (LVLMs) has been significantly shaped by the pioneering success of Large Language Models (LLMs) such GPTs (Radford et al. 2019 Brown et al. 2020: Ouyang et al. 2022) and LLaMA (Touvron et al. 2023 catalyzing advancements in multimodal content understanding and generation (Zhang al. 2024a), including intricate tasks like image-text comprehension. At the forefront of these developments, BLIP-2 (Li et al. 2023d)
introduces lightweight Q-Former (Li al. 2023d) that facilitates alignment between textual and
3

---

Published as a conference paper at ICLR 2025
visual representations through cross attenntn mechanism (Li ct al. 2023dh. InstructBLIP (Dai et al.
2023) takes a step further by incorporating textual instructions into the Q-Former, which significantly improves the zero-shot performance. LLAVA (Liu et all 2023b) employs GPT-4 (OpenAl 2023
to transform data into multimodal instruction-folloviii
data and uses CLIP (Radford et al. 2021
and LLAMA (Touvron al. (023) for fine-tuning instructions, achieving advanced multimodal chat abilities. LLAVA-1.5 (Liu et al. 2023a) extends this paradigm by integrating MLP projection and introducing academic task-specific Vision-language QA data. Recently, models like Otter (Li al. 2023a), MiniGPT-4 (Zhu et al. (023), -L-Ch-t (Bai et al., (023a) and XComposer- VL (Zhang et 2023)) further unleash the cross-modal understanding capabilities of LVLMs. Besides, many powerful closed-source LVLMs, including Gemini- 1.5-Pr (Team et al. 2024) and GPT-4o (Ope).
have publicly released their APIs, promoting the development of downstream applications.
2.2 BENCHMARKS FOR LVLMS
The great progress of LVLMs triggers the development of benchmarks for evaluating these models, where we divide previous benchmarks into three categories. The first type is the classical benchmarks which focuses on evaluating LVLMs abilities via image caption (Lin et al., 2014) and VQA (Antol eet al. 2015b a). However, these benchmarks cannot provide the fine- sine-grined feedback on how to improve the models. Besides, since these benchmarks have been the public resources for long time, it is hardly to guarantee that the LVLMMs have not use them for training. The second type subjectively evaluates LVLMs by experts ( (Yang et al. 2023b] Wu et al. 2023). p. Although these benchmarks reveal the insightful feedback of the LVLMs, their scale is limited (i.e., less than 200 annotations). The subjective manner also makes the evaluation expensive and hardly to expand the scale.
The third type (Liu et al.
2023b Fu et al.,
2023
Xu et al.
2023 Shao et al.
2023 Li et al.
2023c b. Liu et al. [223c, Bai et al. 2023b
Li et al. [2023 Yang et al. 2023a Yu et al. 2023
Ge et al. [023|Car et al.] 2023 Chen et al. 2024 Liu et al.
(022) focuses evaluating LVLMs
in an objective and large-scaled manner, where we list the detailed information of them in the
Tab. [ Some of them have been adopted by the community (Contributors) (023) as the standard
benchmarks for evaluating LVLMs (OpenAï 2022, Zhang et al. 2023). like MME (Fu et al. 2023
and MMBench (Liu et al.) 2023c). These benchmarks evaluate models through the objective answer types and most of them leverage the automatic annotation and evaluation manner for revealing the iinegrriieed drawbacks of current LVLMs. However, the previous benchmarks primarily concentrate on evaluating LVLMs using realistic images under clean scenario. leaving multi-stylized images and noisy scenarios unexplored. Moreover, many of them conduct QA by selecting images from publicly available datasets (e.g., (L.in et al., 2014 [Russakovsky et al., 2014)). While they state that the questions have been re-annotated, they cannot guarantee that the LVLMs have not seen the image during training stage. The previous work (Chen al. (024) has proved that these benchmarks have unintentionally leaked into the training data of LLMs and LVLMs. One possible way to solve data leakage is using novel but synthesis images, where JourneDB (Sun al.. 2023a) is the first work aiming to leverage synthesis images to evaluate current LVLMs. The prompts together with the corresponding images are downloaded from Midjourney (mid) and ChatGPT (OpenAl 2022) is leveraged to label the images. However, JourneyDB is a top-down framework where the number of images is fixed. Besides, the ChatGPT labeling may cause hallucinate annotations, leading to the unreliable evaluation results. In contrast, Dysca serves as a bottom-up framework which enables dynamic and scalable generation of both images and evaluation questions, while also supporting varied evaluation tasks. The rule-based question generation method also makes the annotations more accuracy.
3 DYSCA
3.1
OVERVIEW OF OUR PIPELINE
The overview of our pipeline is shown in Fig. [ containing data generation, data cleaning and LVLMs evaluation. For the data generation, our Dysca benchmark consists of four dimensions, ie.. L, where M means "Metadata", P means "Prompt", I means "Image" and Q means "Question-answer pair". We further decouple the metadata M into 4 parts, i.e., "style", "attribute", "foreground" and "background", and the combination of the four parts constitute the image prompts P. Then, given the prompt P and the selected scenario, leverage the Text-to-Iman Image (T2I) diffusion model (c.g., SDXL (Podell et al., (203)) to synthesis image and add the specific perturbation to the image T. After that, since the prompt already includes the question angle and the corresponding
4

---

Published as a conference paper at ICLR 2025
nos Pedlniion
am dtotts
Table 2: Key statistics of Dysca.
Statistic
#Number
Leviuok
Coler Color-itet 11822
Total questions
617K
8b 2012, More
Ansoul 1181 13
Clean
156K (25.2%)
Currs Gnital
Print attacking
149K (24.1%%
Chalee
Adversarial attacking
156K (25.2%)
Corruption
156K (25.2%)
ofee Oca
Utpici 11114
Perception
Question type
S15s Actin
Auutii-coiiess
251K (40.6%)
Plat- stay
True-or- fralse
250K (40.5%)
Ross Codeis
Fine Granne
Fiee- sdt
Free-form
116K (18.8%)
ezz Siponsie
Image resolution
1024
Unique number of images
289K
Unique number of questions
162K
Unique numher of answers
31K
Average question length
37.8
Figure 2: Overview of the dataset distribution of 20 perceptual tasks. The number in each subtask shows the corresponding amount of their annotation.
Average answer length
2.7
Average choice number
3.0
answer, we construct a rule-based approach to generate the Q. Three types of questions are considered, i.e. multi-choice, true-or-false and free-form. Multi-choice and true-or-false questions utilize closed- ended manner to assess LVLMs, while free-form questions employ an open-ended manner through image captioning for evaluation. For the data cleaning. considering that the T21 diffusion model may generate unsuccessful outcomes, we then use CLIP RRdfford et al.. 2021) and PP-OCRv3 (Li et al.) 2022) to automatically clean the whole dataset to obtain the final Dysca. Finally, evaluate 14 open-sourced LVLMs and 2 closed-source LVLMS on our proposed Dysca.
3.2 PERCEPTUAL TASKS
Evaluation dimensions. Perception is of the most fundamental capabilities of LVLMs and
previous works (Fu et al. 2023) have shown that the lack of perceptual ability may result in halluci- nation (Li al. 2023e). In order comprehensively evaluate LVLMs' perception capability, collect and organize existing sub-dimensions from current benchmarks, resulting in 20 assessment dimensions where show all the subtasks and the corresponding amount of their annotation in the We investigate on two types of perception dimensions, i.e., coarse-grained and fine- tne-graieed perception. Coarse-gre graase-grained perception involves recognizing the style, background and color of images. Fine-grained perception involves recognizing the animal, object, plant, food, age, gender, expression, race, celebrity, action, text, clothes, movie, anime, landmark, profession and TV shows.
Fig. [
Data sources. For each perceptual subtask, we collect the textual data first to construct the metadata M. For the TV shows, Anime and Movie, we select the titles from the rating list of IMDU based on the number of user reviews. For the styles, we utilize the style lists collected from the community and remove those which have strong reflect on the image content like "architectural style" and "Pokemon style". Note that the style list does not include the style prompt associated with a particular artist's name. Besides, for the remaining contents, we select them from the label of current dataset (e.g.. ImageNet (Russakovsky al. 2)144). All the selected textual data above constitute the metadata M. We provide the detailed information of the metadata the Appendix H
3.3 CONSTRUCTION OF QUESTIONS & ANSWERS
Recall that the data generation for Dysca benchmark consists of four dimensions, i.e., P, I, Q), denoting the metadata (M), prompt (P), image (7) and question-answer pairs (Q), respectively. The relationships between these parts and the process of constructing Dysca are shown in Fig. The metadata M is the core of the whole Dysca, containing all the information for generating P. and Q. The metadata M consists of foreground, attribute, background and style, which guides the generation of the prompt (P) through pre-designed templates. Then, we utilize the T2I diffusion model to generate the corresponding image using the prompt P. For generating the image with specific text on it for the OCR subtask, we leverage TextDiffusion2 (Chen et al. 2023a), which is the of
'httss//www.imdd..om htan:://saalee-dfffsiiorarrt.t//
5

---

Published as a conference paper at ICLR 2025
M
Pre-defined Templates
( iPhone photo, large depth field, deep depth of field, highly detalled
Prompt: "white color kettle, lawn, iPhone photo, large depth of field, deep depth of field, highly detailed"
white color
kettle
on the lown
Style
Attribute
Foreground
Background
Type: Multi-chiice
Quention: What is the abjject of
Typac Trup-unSaee
Type: Free form
picture?
bottle
Question:
piaure yellow
Questim:: Pizzac dourbbe inage. Indraction:: You deserieei from those rapasacc vabject the imags color, the barkgrandd artivlli style the image Answer CT: white color kettle, on the lown backgrunn, iplane photographic style
Tesi-t-lmage
ketile
True
D. dining table
Instruction: Please anower the quescon and provide die cornect optiot. loner, g. (A), (R), (C) (P) at the and. Du not contain the undyais peogress Your atswer is:
Instruction: Please at the ypastion
and provade the cornect option letter gg.
(A). (I), (C), (I)) at the end Do
costain the analyis progress Your
unwer
Answer GT: Fibse
Answer GE: kottl
Questions of different subtask and types
Figure 3: The process of generating the prompt (P), image (I) and QA pairs (Q) from metadata (M).
art text rendering method. For the rest of images, we leverage Stable Diffusion XL (Podell et al.. 2023). Subsequently, based on the different question types we select, i.e., multi-choices, true-or-false and free-form, we generate the corresponding VQA pairs in Dysca.
Besides, in order to evaluate the model performance under various scenarios, we conduct experiments on 4 scenarios, i.e., clean, corruption, print attacking and adversarial attacking. For the print attacking. followed by (Cheng et al., 2024), we add the deceptive text on the image, where the text is a wrong option. Besides, to comprehensively evaluate the performance of LVLMs under corruption scenario, we add more typographic factors to original settings (i.e., different font orientations and font positions). For the adversarial attacking. we leverage PGD (Madry et al., 2017) to generate the adversarial image. We use InstructBLIP (Dai et al.; 2023) as the proxy model and regard others the black box models. The reason why we choose InstructBL.IP is that it has shown superior performance in clean scenario. Besides, the black-box setting better reflects the robustness of the models when they face the real-world adversarial attacks. For the corruption, we leverage the image corruption methods collected from (Zhang et al. 2024b). We remove some hard corruptions as they significantly impact the quality of the image, leading to human failure in judging the style and content of the image. The detailed examples are shown in Appendix [F]
Data Clean. To ensure the quality of Dysca, four steps are adopt: 1) First, we manually remove difficult-to- -e-eeertte foregrounds and attributes, along with backgrounds and styles that could heavily affect image content. We believe this process can serve as a coarse- coarse-grained method to eliminate samples that are highly likely to be generated incorrectly. 2) Then, we leverage the off-the-shelf models, i.e., PP-OCRv3 (Li et al. 2022) and CLIP-L-14 (Radford et al.) 2021). to clean the data. PP- OCRv3 Li et al. 2022) is leveraged as the filter to exclude the failure image that TexuDiffusion2 (Chen' al. 2023a) generates the wrong text on the image. For the other images, we use CLIP-L-14 (Radford et al. 2021) with a threshold of 0.75 to filter out the images with low text-image consistency. We find that using 0.75 as the threshold achieves good balance between image correctness and data scale. 3) After that, We select the top six performing models and eliminate any question-answer pairs where the models either answer incorrectly indicate that the answer was not included among the options. We observe that nearly 100% of the samples filtered out by these models are incorrect. 4) Finally, we analyze the patterns in these incorrect samples, removing the associated vocabulary from our metadata and discarding all related samples. By meticulously refining the metadata manually and utilizing automated tools to assist in question filtering, Dysca ensures high- high-quality data synthesis. In the end, we filter out nearly 40% of low quality samples. The final statistics of our released Dysca are shown in Tab. 2 Note that the OCR subtask does not involve print attacking scenario as misidentifying adversarial text does not indicate poor OCR robustness of the LVLMs. Therefore, there are 7K fewer questions in the print attacking scenario. Besides, for the free-form question type, since it allows to assess the model's 's perception abilities across multiple subtasks at the same time, we reduce the number of free-form questions for achieving a balanced data distribution.
3.4 EVALUATION STRATEGY
Instruction Design. We design two types of instructions to improve the instruction instruction-folloiing result of LVLMs. For the multi-choices and true-or-false questions, we design the questions followed by

---

Published as a conference paper at ICLR 2025
[0werien) Is fure core the place! (4) day Time
[Qsedian] luthe peron is piznr playing plano? (A) Iaae Palo
Haaden Loe Tiee (8) Rilse
Rantig
IAALIA
[Gesiss11.]]
[Genini-1.5]
[Qeas-ELChell
(Q CL-n-V-Cluol
(GRMM-N
[GL
JGRMM-9
(GLMNN]]
1Unre-11-Chan (I) Tree
[Essracttil]] (t)
[Gentall (AI Trac
[Gembal] (8)
[GrT-ta]
[GPT--60]
[GPT--6g
(GPE-H6
[GPTT (4)
[GP144] (oke
Figure 4: The failure cases for the noisy scenarios. From left to right are: corruption scenario, adversarial attacking scenario, and print attacking scenario.
MuttiC Belos
MuttiChhttem
[Qucaliee| What enisal in ionge?
Ruwsiesd Qucakin Who i the edélicky innge? (Al Chantes there Cate Blnchett
(Ao pumm
[Madal Amswer] suseris
|Mold Asmwett Bare KQustient dacelldeity pictaze Can. Bandadtf (A) File (8) Tras
(Quesioa) l thacc quazon lie pictadd
(N) True (ake
[Madel Anewer] (10) Eotlo
[Made]] NcorpporrVV.
|Model Ancred
[Moda] Bit2
Figure 5: Models exhibit different performance when facing the same image but different question types.
the description "Please answer the question and provide the correct option letter, e.g., (A), (B), (C), (D), the end. Do not contain the analysis progress. Your answer is: ". For the free-form questions, recalling that the prompt contains four part, i.e., the style, attribute, foreground and background, we instruct the model caption these four dimensions by "Please describe the image. You can describe from these aspects: {)", where "()" includes the specific template we design for each part. We display the sample in the Fig. and more examples can be found in the Appendix
Evaluation Metrics. For the multi-choices and true-or-false questions, we use accuracy as the evaluation metric. We randomly shuffle the order of choices to prevent evaluation results from being influenced by the model's 's tendency towards specific choices (Zong et al. (202)). The random accuracy of the types are equal 25% and 50%, respectively. We use regular expressions extract the model's is answer choices. For cases where the extraction is fail, we calculate the Levenshtein distance between the answer string and the choice string, and select the option with the minimum distance as the model's answer. For the free-form questions, we test the model's 's image caption capability where the ground truth is the prompt of the image. Followed by (Xu et al. 2023), we use pintenccTransformae (Thakur et al.) 2021) to compute the text similarity with prompt P and the caption output of the LVLMs. The final score of each question type is the average score of subtasks.
Besides, thanks to generative evaluation framework of Dysca, we are able to effectively control variables and conduct detailed analysis of the model's fine-grained capabilities. Specifically, we introduce two novel metrics to measure the sensitivity of LVLMMs question types and covariate shift (i.c., image style). The sensitivity to question type aims to evaluate whether LVLMs exhibit inconsistent performance when facing different types of question types (i.e., multi-choice
true-or- false). We first normalize the score of true-or-false by TFSQ FQ = 100-50 100% and multi choices
50
by MCSQ T00-23 100%, where S denotes the score of the models in each question type. The sensitivity of LVLMs to question types is defined by:
$-25
SQ (TFSQ (Qqng)² (MCSQ- -SBIQQQ--QAar =SQgn)"
where SQAur= TFSQ+MCSQ 2
The sensitivity to covariate shift aims to evaluate whether LVLMs exhibit inconsistent performance when facing the same content and question format, but with variations in image covariates. It is defined by:
SC N (S. ASAagg
(2)
N

---

Published as a conference paper at ICLR 2025
Table 3:
Evaluation results of 26 LVLMs, where the darker colors represent better performance. The top 1 result on each column are bolded and the value in brackets is the relative values with respect to the ones in the clean scenario. "PrintAtt" and "AdverAtt" means "Print Attacking" and "Adversarial Attacking", respectively. " the model is under white-box setting.
Model
LLM
Visual Encoder
Blind
Clan
Corruption
Scenarios
AdverAtt
PrimAu
SQ.
SC
4
Vicuns-78
41:34
(6.96)
42:71 1,331
193.6
1.7
MiniGiPT-4
Vicun-
EVA-CLIP VIT-G
35:21
50:17
49.63 (-0.54)
51.77 (-18-40)
47.85
758.
2.5
MinitirT--
L1MM2-78
EVA-CLIP VIT-G
34.77
56.61
55.70 (-B.91)
53.55 (-23.06)
49.78 (-6.83)
1344.5
3.0
MiniGPT-22
LLMMA2-7B
EVA-CLIP VITG
35.28
58.46
58.06 (-0.40)
56.62 (-1.84)
52.96 (-5.50)
1512.1
4.3
BLIP2
Fla-13-XL
EVA-CLIP VIFG
35:35
65.30
60.99 (+0.79)
32.55 (:32.75)
52.01 (-8.29)
165.5
8.5
RI IP2
GPE3B
EVA-CUPP VIT-G
$4.99
50.54
40.29 (+0.75)
(-8.92)
37.26 (-2.28)
110.7
2.5
BLIP2
OPE-78
EVA-LIPP
1.8
InstrucIBLIP
Vicun=
EVA-CLIP
(-0.5))
(-33.113))
1287.1
InstructBLiP
Vicuna-
EVA-CLIP VIFG
34.37
(-0.21)
31.77
53.53 (-11
1252.4
2.5
InstrctRl..n
Flmn-T5-X1.
EVA-CLIP VITHG
34.51
66.54
67.58 (+1
32.95 (-33.599
(-14 44.45)
271.3
6.3
IndruciBLPPP
Fla- XXL
EVA-CLIP VIT4
34.82
08.65
(9:79 (+1.14)
32.95 (-35. 70)
57.75 (-1.992)
215.0
6.8
1Lawa-155
Vicun= 76
CLIP VIT-L
34.63
51.27
51.70 (+0.4))
49 62 (-1.65)
47.27 (-4.00
788.
3.2
LLava:15
Vicuna- 13B
CLIP WILL
35:21
59.23
59.58 (+0.35)
56.87 (-2.36)
51.69 1-7.541
912.5
7.7
Citer
L1aMA-7B
CLIP VIT-I.
$5.19
$4.90
56 (+1.12)
S1 (-3.48)
37.78 (-17. 12)
427.4
6.0
Shikzn
LLaMA-7B
VIT-L
34.96
440.6
Xcomposcr-VI
EVA-CLIP
(-4.1)
147.9
Xcompose2-VVL VL
IntemLM2-1
32.76
79.13
78.64 (00.9)
76.00
(:1.79)
20.9
1,9
Quen-VI.
Quen-78
OpenCTii WTTNig
33.06
61.05 (-1.13)
(-2.35)
$1,94
1885,4
6.7
Eamu2-Chat
LLMMA-33
EVA2-LIP
35.14
63.64
6281 (-0.35)
(-1.74)
54.82 (-8.82)
2497.9
1.6
GL GLMM-Y
GI--4-9B-Cht
EVA2-CLIP-E
35.08
82.09
$1.95 (-D. 14)
H0.72 (-1.37)
$2. 09 (-30.00
25.5
11.8
MiniPPM V2.5
Liamak Instrctt SB
SigLIP SOVIT-400m
34.99
78.75
77.41 ( (-1.34)
(3.31)
(-17.9))
38.3
1.7
Vi-VL
Ys-BB Chat
OpenClip VIT-II
35.01
75.71
74.94 (-0.77)
72.53 (-3.18)
64.97 (31.74)
233.1
1.9
mPLUG-Out-2
LLaMA2-7B
35.03
180.9
2.2
VhiconViion
Phi-3
VIT-L
34:74
(-36-7))
25)
202.7
GPT40
35.02
75.69
(-0.17)
(-2.22)
56.34 (91355)
67.4
1.9
Gemini- 1.5-Pro
34.55
77.79
(-0.77
(-1.9)
(-105(-16.74)
439.6
1.5
where SCAvg ΣN S. and S, denotes the score of the models in each style. Since there 51 image styles in Dysca, we set N = 51.
4 RESULTS AND ANALYSIS
In this section, we report the evaluation results and make insightful analysis. total of 26 LVLMs are evaluated on Dysca, including BLIP2 (Li et al. (023d), InstructBLIP (Dai et al., 2023). LLavA (Liu al. 2023a), MiniGPT-4 (Zhu et al. 2023), Otter (Li et al. 2023a XComposer- VL. (Zhang et al.
2023).
2023b). Emu2-Chat (Sun et al. GLM-4V (GLM et al. 2024). MiniCPM-v2.5 Yao et al. 2024). Yi-VL (Al et al. 2024).
(Bai et al. 2023a). Shikra (Chen et al.)
2024).
mPLUG-Owl-2 (Ye et al. 2023 Phi-3- Phi-3-Vision (Abdin et al. 2024), GPT-4o (OpenAl 2023). Gemini-
1.5-pro (Team et al. (024). Each model is evaluated with all the 20 perception subtasks under
scenarios. The detailed rankings for each subtask can be found in the Appendix A
4.1
MAIN RESULTS
Blind Setting. We first evaluate LVLMs when only textual questions are provided. As shown in the "Blind" column of Tab. 3 all LVLMs yield consistent results on the Dysca and perform comparable to random guessing. This outcome demonstrates that the generated paradigm employed by Dysca
effectively mitigates the potential impact of data leakage (Chen et al. 2024). thereby enhancing the
fairness of the evaluation results. Additional comparisons can be found in Appendix D
Clean Scenario. The evaluation results of various LVLMs in different perceptual subtasks under clean scenarios are presented in the "clean" column of Tab. We calculate the average score of question types. As can be seen, GLM-4V (GLM et al., 2024 outperforms other LVLMs, achieving top-1 performance. MiniCPM-v2. (Yao et al. 2024). Xcomposer2-V VL (Dong et al.) (024) and Seminni-1.--pm (Team et al.) 2024) also perform well. It is evident that for the latest large models, their scores remain below 85. The results highlight that all existing LVLMS still struggle to provide accurate responses to questions formulated by Dysca.
Noisy Scenarios. The evaluation results of various LVLMS under noisy scenarios (i.e., corruption, print attacking and adversarial attacking) are presented in last 3 columns in Tab. 3 The value in the brackets shows the relative values with respect to the ones in the clean scenario. As can be seen, GLM-4V (GLM et al.] (024) still takes a lead on corruption and adversarial attacking scenarios. For the print attacking scenario, mPLUG-Owl-2 (Ye et al., (202)) performs the best. Here, we present a failure case sample for each of the three different scenarios on Fig. 4
8

---

Published as a conference paper at ICLR 2025
4.2 KEY OBSERVATIONS
(1) For LVLMs, the capacity of the language model plays a erucial role. When using the same visual encoder, models that utilize a language model with a larger parameter sizes (e.g., MiniGPPT- achieves an improvement score of 8.79 in the clean scenario when using Vicuna- 13B compared to Vicuna-7B) or with a stronger capability (e.g., GLM-4V that uses the GLM-4-9B-Char language model shows an improvement score of 18.45 in the clean scenario compared to Emu2, which uses the LLaMA-33B language model) tend to achieve better performance.
(2) Each model shows robustness in the corruption scenario, but experiences significant degra- dation in both attack scenarios. In the image corruption scenario, all models demonstrate minimal score variations, i.., less than 1%. However, under print attacks, performance drops are notable. For instance, closed-source models exhibit significant performance degradation: Gemini-1.5-pro declines by 21.5%, resulting in a score reduction from 77.79 to 61 10, while GPT-4o shows a 25.8% decrease, dropping its score from 75.69 to 56.10. Among the leading open-source models, GLM-4V experiences a sharp 36.5% performance drop. lowering its score from 82.09 to 52.09, and Phi-3- Vision records a 21.39% decline, reducing its score from 73.23 to 57.78. Notably, mPLUG-Owl-2 demonstrates the highest robustness, with only 1.7% reduction. The XComposer-Vi VL series also exhibits strong resilience against print-based attacks. In the adversarial attack scenario, where the attack algorithm directly targets the image encoder, LVLMS employing a shared encoder architecture (e.g., Blip2, InstructBL.iP, and XComposer-VL. all of which utilize EVA-CLIP (Fang et al. (022)) as their image encoder) suffer substantial performance declines, with some models performing even worse than random chance. For example, XComposer- VL experiences a 57.6% drop, reducing its score from 71.40 to 30.28. Models with different image encoders also experience performance degradation ranging from 1% to 5%, showing a higher impact than that of corruption noise. More detailed results are available in Appendix FF
(3) Models exhibit varying sensitivity to different question types and covariate shifts. For the sensitivity to question types, we present two examples in Fig. 5 As be seen, the XComposer- VL (Zhang et al. 2023) recognizes the sparrow in the image under a multiple-choice setting but fails to identify the sparrow in the same image under true-or-false setting. The quantity results are shown in Tab. 3 Xcomposer2-VI achieves the best result with score of 20.9. However, we observe that the perception ability of LVLM does not show a positive correlation with sensitivity to question types. For instance, while the GLM-4V achieves the highest performance in evaluation tasks, it exhibits higher sensitivity to question types than Xcomposer2-VL. One of the factors influencing the sensitivity to question types may be the inherent biases of the language model. Using the same base language model may result similar outcomes for this metric. it is also noted that Gemini- 1.5-pro performs bad in this metric, revealing its preference for certain question types. For the sensitivity to covariate shifts, as shown in the last column of Tab. 3 GLM-4V achieves the best result with score of 0.8. However, we also observe that the perception ability of LVLM does not show positive correlation with covariate shifts. For example, MiarructBLIP-Flaaa Flan-FlSSSSS T5-XXL outperforms MnstructBLIP-Flann ----L in terms of performance but shows higher sensitivity covariate shifts.
4.3 ANALYSIS ON INTER-TASK AND INTRA-TASK
Analysis on Inter-task, In order to investigate the inter-relationships across evaluation dimensions, we conduct hierarchical clustering based on the Euclidean distance of the scores across 20 dimensions for 26 models. We observe that models show varied consistency of performance across the dimensions. LVLMs tend perform better in dimensions that involve well-defined image perception, such as landmark recognition and object recognition. However, dimensions likes style recognition and movie recognition exhibit greater challenges for LVLMs, which may be attributed to the limited training resources in these specific domains. Besides, commercial models exhibit poor performance in tasks related to "people" (i.e., "race", "age", and "gender"). This is likely due to the additional safety training incorporated into closed-source models, which results in more conservative responses related questions. Detailed results are provided in Appendix A
Analysis on Intra-task. Thanks to the diverse metadata design, Dysca enables highly granular analysis within a single task. Taking animal categories as an example, we analyze the performance of LVLMs different animals. Here, perform the same hierarchical clustering methods across 51 animal categories. LVLMs performance vary significantly across these categories. We find that models tend to perform poorly when applied to marine life. This could be caused by the challenges

---

Published as a conference paper at ICLR 2025
in collecting icean-related data. This observation highlights the need to direct the model's focus to oceanic domains.
4.4 THE VALIDITY OF DYSCA
In this section, we investigate on the evalu- ation gap between Dysca and non-synthesis benchmarks. We calculate the Spearman's 's rank correlation coefficient (Spearman 1904) and the Kendall rank correlation coeffi- cient (KENDALL) 1938) T between the eval- uation ranking of Dysca under clean scenario with the non-synthesis benchmark's 's evaluation ranking, i.c., MMBench (Liu et al. (023c), OCRBench (Liu et al.
Table 4: The correlation results three bench- marks, where E [-1, and T E 1-1.
Style
Method
MMBench
OCRBench
SeedBench-2
All
P
0.70
0,90
0.46
0.60
0.50
0.43
Realistic
0.70
1.00
0.64
0.60
1,00
0.62
2024) and SeedBench-2
al. 2023b). Both coefficient generate score in the range of [-1,1], where represents a perfect positive correlation, 1 represents a perfect negative correlation, and 0 represents no correlation. These coefficients are typical tools for measuring the correlation between variables statistics. When the absolute value of either coefficient exceeds 0.6, it is considered to indicate significant correlation (Akoglu 2018). Specifically, we intersect our Dysca with current benchmarks based on the perceptual subtasks, evaluation models and evaluation question types. We then calculate the correlation of model evaluation rankings within this intersection. The results are shown in the first row of Tab. For the MMhench and OCRBench, both metrics show the high correlation, with p and r higher than 0.6. However, the correlation for SeedBench-2 is not as strong. Considering that SeedBench-2 only contains realistic images, we conduct additional experiments using the evaluation ranks on our realistic style images only. As shown in the second row of Tab. 4 the correlation results of SeedBench-2 significantly improve (i.e., 0.46 vs. 0.64 for p and 0.43 vs. 0.62 for 7). The correlation with OCRBench also improves to 1, demonstrating the validity of using synthetic datasets for evaluation LVLMs.
Besides, we calculate the data distribution distance between each benchmark to prove the low distance distribution between Dysca and non- synthestt benchmarks. We select CCBench (Liu et al. 2023c), COCO-Va Val (Lin et al. 20144), MMVet (Yu et al. 2023
0.14
E231
8.15
0.3
1:24
922
05
MMBench (Liu et al.) 2023c MME (Fu et al., (023). MM-
6.21
0.2
6:21
Star (Chen et al., 2024), OCRBench (Liu et al.) 2024) and Sci-
enceQA (Lu et al. (002)) The reason why choose these bench-
A21
8.51
6.55
mark is that they have been widely used in evaluating LVL.S. We Kernel Maximum Mean Discrepancy (KMMD) (Schölkopf iet al., (207)) to measure the distribution distance. Specifically, we randomly sample 3,000 images from each benchmark (if the scale of the benchmark less than 3000, we use all that data) and utilize CLIP (Radford et al. 202T) to encode these images. Then, we calculate the KMMD value using an RBF kernel between each pair. The results are shown in Fig. 6 Each row represents the value of KMMD between two benchmarks. The value of last row denotes the average value of KMMD. As can be seen, the distribution distance between Dysca and real-image benchmarks ranks in the middle compared to all other benchmarks, indicating that the evaluation results can effectively reflect the model's 's performance in real-world scenarios.
8:22
012
6.29
8J00
0.5
621
8.25
6:45
6:55
0.15
6.35
8.00
6.28
8.55
0.15
1.25
8.16
0.25
6.35
340
Figure 6: The KMMD distance between each benchmarks, with darker colors indicating larger distances.
5 CONCLUSION
In this paper, we purpose Dysca, a dynamic and scalable benchmark for evaluating perception ability of Large Vision Language Models (LVLMs). Dysca consists of 617K Vision-language QA pairs, covering 20 perceptual subtasks, 4 image scenarios and question types. We conduct the experiment on 24 advanced open-source LVLMs and closed-source LVLMs, revealing the insightful weakness of current LVLMs when facing different question types, image styles and image conditions. Experiments demonstrate the validity on evaluating LVL.Ms by using synthesis images.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is partially supported by Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0680202), Beijing Nova Program (20230484368), Suzhou Frontier Technology Research Project (No. SYG202325), and Youth Innovation Promotion Association CAS.
REFERENCES
Openai. hello gpt-4o. httpp:://peeat..ccomm///eeee//// 40/
Join the midjourney discord server! httpp///dddserrd..oo////eeeeeees
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, and Nguyen Bach et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint
0et::2404.122111 2024.
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, and Guanwei Zhang et al. Yi: Open foundation models by Ol.ai. arXiv preprint arXiv:2403.0445 2024.
Haldun Akoglu. User's guide to correlation coefficients. Turkish Journal of Emergency Medicine, 18 (3):91 -3,, 2018. ISSN 2452-2473.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2425-2433, 2015a. doi: 10.1 10/9/CC
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2425-2433, 2015b.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vasion-language model for understanding, localization. text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023a.
Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang. Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308. 16890, 2023b.
Praneeth Bedapudi. Neural nets for nudity detection and censoring, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 13:1877-1901, 2020.
Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, and Alex Kot. Benchlmm: Benchmarking cross-style visual capability large multimodal models. arXiv preprint arXiv:2312.02880 2023.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power language models for text rendering. arXiv preprint 1rXiv:2311.16466 2023a.
Keqin Chen, Zhao Zhang. Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's referential dialogue magic. arXiv preprint 1iXi:22006.11955 2023b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. we on the right way for evaluating large vision-language models? arXiv preprint 2rXi:22403.20330 2024.
Xinlei Chen, Hao Fang, Tsung- Yi Lin. Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: /rX4v:/504.000 150..00344...... 2015.
11

---

Published as a conference paper at ICLR 2025
Hao Cheng, Erjia Xiao, Jindong Gu, Le Yang, Jinhao Duan, Jize Zhang, Jiahang Cao, Kaidi Xu, and Renjing Xu. Unveiling typographic deceptions: Insights of the typographic vulnerability in large vision-language model. arXiv preprint arXhv:2402.191500 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction instruction-finetueed language models. arXiv preprint arXiv:2210.11422 2022.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. htttps://githii http://ttthu...o.pppe.e..ee cowpass//ppnncopaaaatttee 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs. vision-language 2023.
models with instruction tuning. arXiv preprint:2305.066
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Ieee, 2009.
Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang, Linke Ouyang, and Xilin Wei Internim-xcomposee xcomposer2: Mastering free-form text-image composition and comprehension in
et al. Internim vision-language large model. arXiv preprint arXhv:2401.16420, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Yuxin Fang. Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits masked visual representation learning at scale. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 19358-19369, 2022.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19358-19369, June 2023.
FastChat. Vicuna.
2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Rohit Gandikota, Joanna Materzyáska, Jaden from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision (CVPR), 2023.
and David Bau. Erasing concepts
Wentao Ge, Shunian Chen, Guiming Chen, Junying Chen, Zhihong Chen, Shuo Yan, Chenghao Zhu, Ziyue Lin, Wenya Xie, Xidong Wang, Anningzhe Gao, Zhiyi Zhang, Jianquan Li, Xiang Wan, and Benyou Wang. Mllm-bench, evaluating multi-modal I1ms using gpt-4v, arXiv preprint arXiv:2311.13955 2023.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets. arXiv preprint arXiv: 1803::800300011 2021.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, and Diego Rojas et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
12

---

Published as a conference paper at ICLR 2025
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6325-6334, 2017. doi: 10.11099CVR.22017..777
David Ha and Douglas Bek. neural representation of sketch drawings. International Confer- ence on Learning Representations, 2018. htw.l:ttss////tm pttttps://opereviiee ew. net
Hy6GHpkCW
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hamnaneh Hajishirzi, and Ali Farhadi. A diagram worth dozen images. In Bastian Leibe. Jiri Matas, Nicu Sebe. and Max Welling (eds.), European Conference on Computer Vision (ECCV), pp. 235-251, Cham, 2016.
M. G. KENDALL. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 06 1938. ISSN 0006-3444. doi: 10.1093/hiomet/30. 1-2.81.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang. and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprimt:2055.03726 2023a.
Bohao Li, Yuying Ge, Yixiao Ge. Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench-2: Benchmarking multimodal large language models. arXiv preprint urXiv:2311.111 17092, 2023b.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023c.
Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang. Yongkun Du Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, Dianhai Yu, and Yanjun Ma. Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system, 2022.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference Machine Learning (ICMI.), 2023d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large ussionllanguag models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings the (EMNLP), 2023e.
the 2023 Conference on Empirical Methods in Natural Language Processing
Zejun Li, Ye Wang, Mengfei Du, Qingwen Liu, Binhao Wu, Jiwen Zhang, Chengxing Zhou, Zhihao Fan, Jie Fu, Jingjing Chen, Xuanjing Huang, and Zhongyu Wei. Reform-eval. Evaluating large vision language models via unified re-formulation of task-oriented benchmarks. arXiv preprint 2rXiv:2310.025660 2023f.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak. Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab. Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang. Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. arXiv preprint wwX.v:2211.091111 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Lanyy Zimick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), September 2014.
13

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744,222 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 2r01 08485, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 23Xx:23377.06811 2023c.
Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895. 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 2022.
Pan Lu. Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024a.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept crasure in diffusion models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6440, 2024b. doi: 10.110/CVPK5277322044.0066
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.06083, 2017.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019 IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 31 3190-3199, 2019. doi: 10. 1 109//P... 2019.00331.
Meta. 1lama3. httpg/ghtwb.oooo//tmtlllaaa/a//a
2024.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2/03. 14749, 2021.
OpenAI. Introducing chatgpt. https///openaaccomm//tgggudttttp 2022.
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems (NeurIPS), 15:27730-27722 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 2xXi::2377.01522 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning (ICML), pp. 3748-8763. PMLR, 2021.
Javier Rando, Daniel Paleka. David Lindner, Lennard Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:22/0.044 2022.
14

---

Published as a conference paper at ICLR 2025
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. In 2014 IEEE International Conference Computer Vision (ICCV), 2014.
Bernhard Schölkopf, John Platt, and Thomas Hofmann. A kernel method for the two-sample soo-spple--poblemm. In Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference. pp. 513-520, 2007.
Wenqi Shao, Yutao Hu, Peng Gao, Meng Lei, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, and Ping Luo. Tiny Ivlm-ehub: Early multimodal experiments with bard. arXiv preprint arXiv:2308. 2tt.:23300.03799. 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3309-8318, 2019. doi: 10.1100//
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):22-101, 1904. ISSN 00029556.
Yixuan Su, Tian Lan, Huayang Li. Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang. Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 49659-49678 Curran Associates, Inc., 2023a.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models in-context learners. 2023b.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, lean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, and Anja Hauth et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:312.11800 2024.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 296 310. Association for Computational Linguistics, June 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin. An carly evaluation of gpt-4v(ision). arXiv preprint arXiv:2310.16534 2023.
Peng Xu, Wengi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lylm-ehub: comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.098 2306.09666... 2023.
Xiaocui Yang. Wenfang Wu, Shi Feng, Ming Wang, Daling Wang. Yang Li, Qi Sun. Yifei Zhang, Xiaoming Fu, and Soujanya Poria. Mm--iggeench Evaluating multimodal models on multimodal content comprehension tasks. arXiv preprint arXiv22310.09036, 2023a.
15

---

Published as a conference paper at ICLR 2025
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang. Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn Imms: Preliminary explorations with gpt-4v(ision). arXiv preprint 130Y:22309.17211, 2023b.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.042555 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 2023.
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9556-9567, 2024.
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-Ilms: Recent advances in multimodal large language models. arXiv preprint arXIv:2401.13601, 2024a.
Jiawei Zhang, Tianyu Pang, Chao Du Yi Ren, Bo Li, and Min Lin. Benchmarking large multimodal models against common corruptions. arXiv preprint arXiv:2401.1194. 2024b.
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding. Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang. Wei Li, Jingwen Li. Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. mccoppln-coompser. A vision- laniuae large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 2309.11912111 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. arXiv preprint 1rX1v:2310.01651, 2023.
16


Reviews:
[{"review_id": "8xA7VtMsdr", "reviewer": "Reviewer_ZY88", "summary": "This work propose a large-scale dataset, i.e., Dysca, for evaluating the perceptual capability of LVLMs. To build such a large-scale dataset, the author propose a flexible pipeline which defines the metadata and a rule-based approach for generating the text-to-image prompt, image and question-answer (QA) pairs. In summary, Dysca consists of 617K Vision-language QA pairs, covering 20 perceptual subtasks, 4 image scenarios and 3 question types. The author conduct comprehensive evaluation for 24 existing LVLMs on Dysca, demonstrating the weakness of existing LVLMs.", "questions": "It is very common that the text and image cannot be aligned well for T2I generation. As far as i know, only the data clean part in the pipeline is related to this problem. How do you ensure the threshold and biding (top 6 models) way can work for it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed dataset is comprehensive and large in scale. Dysca not only considered different perceptual scenarios, it also provided a testbed for evaluating the perceptual capability of LVLMs under both corruption and printing attack situations.\n2. The data construction pipeline is flexible to be extended for assembling the samples of other sub-tasks. Benifit from the powerful generation capability of T2I models, the pipeline also demonstrated the good quality of generative large-scale datasets.\n3. The experiments are comprehensive and proved the claimed merits of the proposed dataset.", "weaknesses": "There are two main concerns:\n1. No data leakage is one of the main claimed contribution for the dataset, but there is no comprehensive statistical comparison of Dysca with existing benchmarks. Only the blind experiment in Appendix D, which should be moved to the main content considering its importance, trying to explain this point which is not enough from my perspective.\n2. The setup of adversarial attack, i.e., only PGD, is too simple. Compare to the corruption and printing attack setup, only PGD is considered for adversarial attack is not convincible, especially considering there are bunch of emerging studies on the adversarial attack of LVLMs.\n\n\nThere are also several unclear points,\n1. There is no description of the rule-based approach adopted when getting the prompt and QA from the metadata.\n2. No explanatio of the metadata itself. Why the metadata is combined with the four aspects? Is there any dataset- or task-specific relationship when considering this?\n3. It is unclear that how many examples for each type of corruption and printing attack in the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O58dELeeNb", "reviewer": "Reviewer_DRDs", "summary": "This paper proposes Dysca, a dynamic and scalable benchmark that uses large language models (LLMs) and text-to-image diffusion models (T2I) to generate synthetic images, questions, and answers. They filter the low-quality examples by leveraging off-the-shelf models. It is dynamic and scalable. Prior benchmark might have potential data leakage issues and scalability limitations. Evaluation experiments show their findings, such as inconsistency in question types and the vulnerability of attacks. It shows the limitations of current vision language models (VLMs).", "questions": "1. Would authors change \"\" to the right double quotes?\n2. Would authors fix the image captions to be below the image and the table captions to be above the table? i.e., Table 2, Figure 6.\n3. In Section 4.2.2, is there any reference that a correlation is high if it is higher than 0.6?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Their strengths come from the idea of automatically synthesizing dataset curation. The metadata contains information to generate prompt, image, and question-answer pairs. \n1. Scaling is easy because of adopting the synthetic data.\n2. They propose the findings, the importance of the language model, the inconsistent performance to different question types, the difference perceptual performance on different sub-tasks, and the degradation in attack. \n3. They demonstrate the validity of Dysca in the section 4.2.2.", "weaknesses": "Their weaknesses come from the idea of simply scaling data. Note that \"simple\" does not mean \"easy.\" Papers such as [R1], [R2], and [R3] consider the specific points of cultural understanding, Humor, and hallucination. It would be useful to have specific evaluation key points like these benchmarks.\n1. How does the proposed benchmark differ from unifying existing benchmarks into one benchmark? Does it evaluate the model behavior that existing benchmarks fail to cover? Combining the existing benchmarks proposed in Table 1 creates a benchmark covering a wide range of perceptual tasks and question types. Therefore, it should clearly show something that existing benchmarks have not been able to do. For example, it would be valuable to show researchers the characteristics of the samples that all VLMs get wrong.\n4. What specific insights can the proposed benchmark provide that are not observable through existing benchmarks? It would be interesting to propose a new metric that is only available in the proposed benchmark. This is a weakness that can be linked to Weakness 1. Metadata is an important role in sample generation. Similar to [R4], an explainable metric using that metadata could be an example. Figure 7 may be good for fine-grained analysis, but showing 51 styles or considering other meta data at once is difficult to understand at once.\n5. With the growing body of research on training with synthetic data, suppose we follow the training approach: (1) generate data using the same process with Dysca, and (2) train VLMs. Afterward, how should we handle potential data leakage due to the similar data distribution compared to Dysca? It can be useful to know the sub-populations where all VLMs underperform, even when trained on synthetic images.\n7. T2I diffusion models have the limitation of creating all possible images corresponding to question-answer pairs. Could authors discuss this limitation? For example, benchmarks that consider specific vulnerabilities of VLMs [R1, R2, R3, R5, R6] seem to be hard to generate automatically. \n\n[R1] Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest, ACL'23\n\n[R2] Benchmarking Vision Language Models for Cultural Understanding, EMNLP'24\n\n[R3] BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models, ECCV'24\n\n[R4] Attribute Based Interpretable Evaluation Metrics for Generative Models, ICML'24\n\n[R5] NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples, NeurIPS'24\n\n[R6] Vision language models are blind, arxiv'24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l9K8Qe0dco", "reviewer": "Reviewer_3C5q", "summary": "This is a data paper. The authors introduce Dysca, a new benchmark for evaluating Large Vision-Language Models (LVLMs) from the perspective of assessing multi-stylized images and noisy scenarios, as well as test environments involving corruption, print attacks, and adversarial attacks. Dysca uses SDXL images to dynamically synthesize varied image styles and questions, enabling evaluation across 51 styles and 20 subtasks under four scenarios: clean, corrupted, print attacked, and adversarially attacked. They also provide interesting insights into LVLMs' capabilities and performance under diverse conditions.", "questions": "1. Can the authors present the complete prompts used for image generation in the appendix? Could some prompts be displayed?\n\n2. Did the authors assess the quality of automatically synthesized images? How is quality ensured when generating such a large scale of images, especially those containing text?\n\n3. How were the QA pairs (text) constructed on such a large scale? Was ChatGPT utilized? or just all by rule-based template?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The reviewer appreciates the authors' attempt to address a unique aspect of LVLM benchmarks: the evaluation of multi-stylized images and noisy scenarios. This perspective is intriguing and relevant.\n   \n2. The paper is well-written with clear motivations and a logical structure. The presentation is visually appealing with sophisticated diagrams. The detailed introduction of the benchmark data facilitates reader comprehension.\n   \n3. The experimental analysis provides several valuable insights. Notably, there is a high correlation between their evaluation rankings and those of non-synthetic benchmarks, highlighting deficiencies in current LVLMs when facing various question types, image styles, and scenarios.\n\n4. The overall effort in the work is solid. Using an automated synthesis method to generate an evaluation benchmark is an unconventional but intriguing and worthwhile exploration.", "weaknesses": "1. The major concern with this dataset is the use of image generation tool for automatic vision creation. For such images to serve as a gold standard for MLLM evaluation, they must meet a critical assumption: the generated images are perfect and error-free. Unfortunately, even today's state-of-the-art SDXL models, which can produce high-quality images, still blend real and synthetic styles rather than producing wholly realistic images. As humans, we can typically recognize diffusion-synthesized images. SD models excel at creating artistic-style images, but for sure, non-realistic images inevitably introduce bias in evaluations on realistic images. Also, many automatically synthesized images that contain scene texts are often of low quality, i.e., containing inaccurate scene text. Such biases could largely skew model evaluations.\nTherefore, the reviewer suggests that the authors add ample discussion regarding this issue. For example, the authors could include some in-depth analytical experiments to validate the concerns raised above.\n\n\n2. The entire process is fully automated, including the use of CLIP for automated quality assessment of generated QA image-text pairs, without any human inspection, which is problematic.\nThe reviewer suggests that the authors analyze the impact of all automatically constructed data on quality, or consider using a more reasonable evaluation method for assessment.\n\n3. While using automatically synthesized images significantly reduces annotation costs, as a testing set, it is a dubious claim without (or with less) real value. Conversely, generating a large-scale training set through automated image synthesis is undoubtedly positive.\nTherefore, the reviewer hopes that the authors can further explain and clarify this matter.\n\n\n\n4. The main focus of this data should be on testing LVLMs' performance in multi-stylized and noisy images and in scenarios involving corruption, print attacks, and adversarial attacks. If a LVLM benchmark is limited to these aspects, its utility and contribution are significantly diminished.\nTherefore, the reviewer hopes that the authors can further explain and clarify this matter.\n\n\n5. \"_Although they claim that the questions are re-annotated, previous work (Chen et al., 2024) has demonstrated that these benchmarks have unintentionally leaked into the training data of LLMs and LVLMs_\": This claim might largely be inaccurate, as Chen et al. (2024) only discussed very early LVLM benchmarks, which were simplistic and not re-annotated. However, most current benchmarks have re-annotated QA texts, almost solving the data leakage issue.\nPlease provide further details.\n\n6. The authors claim a large dataset size with 617K visual-language QA pairs. However, for evaluation benchmarks, sheer size is not as important as having a more varied set of subsets assessing different aspects and capabilities [1]. Unfortunately, Dysca only covers 20 image perception subtasks, offering no significant advantage over SEED-Bench2 and BenchLMM. Moreover, it appears to be surpassed by existing work (e.g., [2]) in both data quantity and evaluation scope, which the authors have not compared or mentioned.\nThe reviewer suggests adding more comparisons for this.\n\n\n7. As a benchmark, it evaluates too few (comparatively) LVLMs, limited to just a small subset of models (actually 16; the 26 LVLMs are somewhat overclaimed). There are many more state-of-the-art multimodal LLMs that should be included at least in the appendix. the reviewer suggests the authors include evaluations of more models next.\nThe reviewer strongly recommends adding more MLLMs for experimental comparison.\n\n\n\n## Summary\n\nOverall, if the authors had focused solely on evaluating multi-stylized images and noisy scenarios, it would have been a safer claim, even with automated image synthesis. However, by comparing it to current general-purpose LVLM benchmarks, the limitations and significant issues of automated image synthesis are exposed, particularly its inability to replace the importance of realistic images. This is my key suggestion for the authors.  \nThus, \"_We demonstrate for the first time that evaluating LVLMs using large-scale synthetic data is valid_\" might be an overclaim, at least not sufficiently and safely proved in this paper.\n\nOverall, the reviewer is open-minded. If the authors can actively and effectively address the concerns with more discussions, the reviewer would consider raising my rating.\n\n---\n\n[1] Are We on the Right Way for Evaluating Large Vision-Language Models? 2024.\n\n[2] MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks. 2024\n\n-----------\n\nAfter the reviewer-author interaction, the reviewer decided to increase the evaluation to positive as the authors provided enough detailed responses and clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)