Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
Authors: Yue Yang
Shuibo Zhang
Kaipeng Zhang
Yi Bin
Yu Wang
Ping Luo
Wenqi Shao
Venue: ICLR 2025 Oral (2025)
Keywords: Dynamic Evaluation
Vision-Language Bootstrapping
data contamination
Flexible Complexity
Large Vision-Language Model
Primary area: datasets and benchmarks
TL;DR: We develop the first dynamic multimodal evaluation protocol with flexible complexity via Vision-Language Bootstrapping.

Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across multimodal tasks such as visual perception and reasoning, leading to good performance on various multimodal evaluation benchmarks. However, these benchmarks keep a static nature and overlap with the pre-training data, resulting in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce a dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MMBench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.

Published as a conference paper at ICLR 2025
DYNAMIC MULTIMODAL EVALUATION WITH FLEX- IBLE COMPLEXITY BY VISION-LANGUAGE BooT- STRAPPING
Yue Yangl.2,, Shuibai Zhang**" Kaipeng Zhang Yi Bin', Yu Wangl Ping Luo2+t, Wenqi Shao School of Artificial Intelligence, Shanghai Jiao Tong University Shanghai AI Laboratory Tongji University "The University of Hong Kong
ABSTRACT
Large Vision- SaniunLa Models (LVLMs) have demonstrated remarkable capabil- ities across multimodal tasks such visual perception and reasoning, leading good performance on various multimodal evaluation benchmarks. However, these benchmarks keep static nature and overlap with the pre-training data, result- ing in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MM- Bench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.
INTRODUCTION
Large Vison-Language Models (LVLMs) (Achiam et al., 2023 Lu et al.) 2024b Research 2023 Chen et al.) 2024b Liu et al. 2024a) have achieved unprecedented performance across a wide range of multimodal tasks such visual perception (Goyal et al. 2017) and commonsense rea- soning (Zellers et al. 2019). The impressive progress necessitates the creation of nuanced evalua- tions to track LVLM development and understand the capability boundary of LVLMs. It leads to the advancement of a series of evaluation benchmarks such as SEEDBench (Li et al., 2023), MM- Bench (Liu et al.) 2023), and MME (Fu et al. 2023). The evaluation results are crucial in selecting suitable LVLM for various applications.
Despite the proliferation of LVLM evaluations, there increasing concerns about the genuine ca- pabilities of LVLMs (Laskar et al.) 2024), largely due to two key challenges associated with current evaluation benchmarks. 2) Data contamination. LVLMs are pre-trained on large datasets, often sourced from the internet. Unfortunately, many evaluation benchmarks are constructed from similar sources, leading to a high likelihood of overlap with training data, thus causing data contamina- tion (Touvron et al. 2023 Chen et al. 2024a). as illustrated in Figure a and detailed in Section It raises critical concern: "Does the model genuinely perceive and understand the input, or merely memorizing it?" 2) Static dataset with fixed complexity. As shown in Figure [b), existing benchmarks for LVLMs are manually collected (Xu et al. 2023] Li et al. 2023). Once constructed, they are static with fixed complexity, making them inadequate to keep pace with the rapid de-
Equal contribution,f i Corresponding Authors

---

Published as a conference paper at ICLR 2025
Static Evaluation
Training Set
Evaluation Set
Flaed Complaity
vodde
Brlckk
Homan
Kilaaa pos aeaTTIn
VS
CC3M
Dynamic Evaluation
Easy
B
FAURED
druadiy Prosgimts elocit
Dymamik
LAION-40OM
Flaitie Complecty
coco-caption
Lnegrietie
Banck
Rhar dhe
PARS
MiViiceess (Miki
Dyromie Benchhookk
(a) Data Contamination
(b) Dynamic Evaluation vs Static Evaluation
Figure 1: (a) shows that some images in evaluation sets can be exactly found in the training set and their corresponding questions can be solved by the captions of similar training images. (b) compares our dynamic multimodal evaluation with the previous static evaluation. We can see that dynamic evaluation can create various variants upon static benchmarks with flexible complexity.
velopment of LVLMs. To accurately assess LVLM performance boundaries, dynamic, automated
evaluation protocol with adjustable complexity is urgently needed.
However, it is challenging develop dynamic evaluation framework for LVLMs. Existing dy- namic evaluation protocols apply to only Large Language Models (LL.s)). For example, Dy- Val (Zhu al. (203)) dynamically synthesizes test samples for LLMs based on directed acyclic graphs to combat data contamination. NPHardEval (Fan et al. 2023) generates new test samples for NP-hard math problems. Yet. the reliance on graph structure and math knowledge limits the applicability of these methods. Recently, MPA (Zhu et al.) (0244) dynamically creates new ques- tions from the old ones for evaluating LLMs. However, building visual dynamics based on existing benchmarks requires manipulating the visual contents in the image while maintaining its essence, which poses a great challenge and remains unexplored.
In this work, we develop a dynamic multimodal evaluation (DME) protocol with flexible complex- ity by proposing Vision-Language Bootstrapping (VLB). VLB consists of multimodal bootstrap- ping module and judge module. The multimodal bootstrapping dynamically creates new visual question-answerte (VQA) samples through various image and language bootstrapping strategies. These carefully designed strategies do not rely predefined rules, making them applicable to var- ious multimodal tasks. Meanwhile, the judge ensures that the newly generated samples maintain consistency with the original ones, e.g. preserving the correctness of the answer after bootstrapping.
To dynamically create new evaluation suites with flexible complexity, we propose various boot- strapping strategies with complexity control for both image and question. These strategies stem from practical situations in real user interactions, primarily simulating user diversity in two aspects: different visual attention and linguistic understanding. By bootstrapping VQA sample with various transformations flexibly, we can build series of dynamic variants with diverse complexities upon an existing benchmark. Experimental results show that the data contamination issues can also be remarkably reduced on these dynamic variants. Therefore, the proposed VLBB can provide compre- hensive and robust assessment LVLMM capabilities, ensuring that the evaluation co-evolves with the ever-evolving abilities of LVLMs.
Our VLB is general enough to applied to various existing benchmarks. In experiments, we em- ploy VLB to bootstrap several representative evaluation benchmarks such as SEEDBench (Li al.) 2023) and MMbench (Liu et al. 2023). We evaluate various LVLMs including closed-source APIs (GPT-40 OpenAl 2024). Claude3-S Soudee--Sonet (Anthropic, 2024 and 9 popular open-source LVLMs such as DeepSeck- VL (Lu et al. 2024a) and Yi-VL Young et al. 2024). The takeaways of our key
findings follows:
2

---

Published as a conference paper at ICLR 2025
All LVLMs exhibited performance decreases our multimodal dynamic benchmarks, in- dicating existing limitations in current LVLMs towards adapting to different users with various identities. (Section 5.2 and 5.3
Through image bootstrapping strategies, we observed that distracting or focusing visual attention on the image will affect L.VLM'' capabilities to understand the key image content related to the corresponding question. (Section 5.2)
By employing language bootstrapping strategies on existing benchmarks, we found that current LVLMS exhibit increasing sensitivity as modifications escalate from words to sen- tences and finally to context changes. (Section 5.2)
VLB can provide multiple dynamic benchmarks with flexible complexity via multimodal and multi-strategy composition. By combining more difficult strategies, we can increase the complexity of existing benchmarks, posing significant challenges for LVLMs, including advanced models like Intern and GPT-4o. (Section
LVLMs exhibit varying performance changes tasks, especially sensitive to 'Instance Interaction', 'Text Understanding', and 'Spatial Re- lation' tasks. (Section 5.4)
dynamic strategies across various
Overall, the contributions of this paper are summarized as follows. 1) We delve into the data contamination issue of existing multimodal evaluation benchmarks and find a pronounced over- lap between evaluation samples and pre-training data, which hinders the assessment of previous static benchmarks. 2) We propose a dynamic multimodal evaluation protocol called vision-language bootstrapping (VLB). VLB evolve existing benchmarks with visual and linguistic dynam- ics, obtaining various variants with flexible complexity and reduced data contamination. 3) We perform comprehensive evaluations on variety of popular LVLMs, indicating that the existing LVLMs still struggle to adapt to different user interactions and intent. Our code will be available at https://gåthur: woom/yangyuesl14/DDf
RELATED WORK
Data contamination. Data contamination raises widespread concerns across evaluation benchmark datasets in both LLMs and LVLMs. Study (Dodge et al.) (02T) reveals exact match contamina- tion rates ranging from 2% 50% across various benchmarks when compared to C4 pretraining data. Llama-2 (Touvron et al. 2023) reported over 16% of MMLUU examples are contaminated, where 11% are seriously contaminated. Subsequently, the concern of data contamination extends to LVLMs. Recent work (Chen et al. (024a) reveals the data leakage issue in LVLMs, they find some samples are leaked into LLMs' training corpora can be "recalled" with the textual questions and answers directly. For example, Sphinx-X-Mor (Gao et al. 2024) gets 43.6% on MMMU (Yue) jt al, 2024) without accessing images, surpassing its LLM backbone with 17.9%. But it is limited to the textual modality of LVLMs, leaving image modality unexplored and the specific contamination rate undetected. In section 3 we first verify a high rate of contamination in both visual and linguis- tic modalities, between static multimodal benchmarks and training sets. This severe contamination necessitates the development of dynamic evaluation to advance LVLMs more accurately.
LVLM Evaluation. With the rapid development of LVLMs, various benchmarks are proposed for evaluating LVLMs. Early single-task benchmarks, like ok-VQA (Marino et al.) 2019). and MS- COCO (Chen al. 2015), are proposed evaluate coarse-grained ability. Later, current LVLM evaluation benchmarks aimed to provide relatively holistic evaluations for the overall reasoning capabilities of LVLMs, such Lub (Xu al.] 2023), MME (Yin et al., 2023). SEED- Bench (Li et al., 2023), MM- YYu al. 2023). MMBench (Liu et al.) 2023). and Llavebench (Liu et al. 2024b). However, the static nature of current benchmarks makes them susceptible to contam- ination and restricts them fixed level of complexity. Thus, there is an urgent need for dynamic evaluation methods to enable comprehensive assessments of LVLMs.
Dynamic Evaluation. Recently, some researchers have pioneered the exploration of dynamic eval- uation. Study (Zhu et al.) 2023) proposed Dy Val to dynamically generate test samples based on the graph structure to combat data contamination. Similarly, NPHardEval (Fan et al. 2023) generates new evaluation samples for NP-hard mathematical problems. To extend dynamic evaluation to more diverse NL. tasks, Zhu al. (2024a) further developed MPA, which employs L..e-based agents
3

---

Published as a conference paper at ICLR 2025
04.46
33.29
38.18
29.28
29.56
11:19
14.80
21.63
.18.72
1,4
1.54
6.27
LAION (100M)
CCMM (3M)
COCO-Caption
LAION (100M)
CCBM
(9.1M)
Only Contamination
Text Contamination
Figure 2: (a) Existing benchmarks have severe overlap on images with pre-training data. (b) Ques- tions of the contaminated evaluation image also be solved by the caption of similar images from the training set.
transform existing problems into new ones. However, these dynamic evaluations only focus on LLMs and NLP benchmarks, leaving the dynamic evaluation of LVLMs unexplored. Previous multi-modal benchmarks (Shah et al.) 2019 Gokhale et al., 2020; Selvaraju al. 2020) have made attempts to rephrase VQA questions, but they can't 't be automatically applied to various formats of existing benchmarks. Our work is the first to design strategies for bootstrapping both visual images and language questions, to achieve dynamic evaluation LVLMs. Based on an original benchmark, we can generate multiple variants with flexible complexity and lower contamination rates.
3 REVISITING DATA CONTAMINATION IN EXISTING BENCHMARKS
We explore two types of data contamination in multimodal evaluation benchmarks. 1) Image-only contamination. We aim to detect how many images in the benchmark can be found in the pre- training data. To this end, we utilize the CL.IPScoe (Hessel et al. (021) to measure the similarity between images from the evaluation and training set. In our pilot experiments, we find that the CLIPScore between two images exceeds 0.9, it indicates high visual similarity. Thus, adopt 0.9 as the threshold to determine visual contamination. The image-only contamination rate calculated as the ratio of the number of contaminated images and the number of total images in the evaluation set. 2) Image-text contamination. Beyond images, the question and answer of benchmark can also be contaminated. We extend ideas from NLP detection works (Li et al.) (2024)) to identify this image-text contamination. For contaminated image pairs, we determine the question and answer contaminated if the answer can be directly inferred from the captions of the training image. In practice, we leverage GPT-4 to conduct this process.
As illustrated in Figure we examine the two types of data contamination between three popular evaluation benchmarks: SEEDBench, MMBench, MME, and three widely used pre-training sets: LAION-100M (Schuhmann al. 202I), CC3M (Changpinyo al., 202I). COCO-Caption (Chen et al. 2015). The results reveal that each evaluation benchmark presents certain contamination rates across training datasets of various sizes, even with some reaching as high as 84.46% (image-only) and 33.13% (image-text).. Note that the actual size of pre-training data far exceeds our detected maximum of 100M. which indicates that the actual contamination issue could be even more severe.
4 DYNAMIC MULTIMODAL EVALUATION
This section introduces our dynamic multimodal evaluation framework, Vision-Language Bootstrap- ping (VLB). Section 4.1 gives an overview. Section 4.2 and 4.3 present one component of VLB, multimodal dynamics: mage bootstrapping and language bootstrapping. The other component, the judge module, is in Section 4.4 The compositional bootstrapping is described in Section 4.5
4.1 OVERVIEW
Formulation of Dynamic Multimodal Evaluation (DME). Let us begin with defining a dynamic multimodal evaluation for LVLMs. Given an original VQA sample F, (T, Q, A) where 7, Q
4

---

Published as a conference paper at ICLR 2025
Ga?
Visaal Attertion
My Buram Duttia Ofyers
Vauiila
Beuand
Sangle
lnnmaction
Mood Daiathdii
Queckal
Add Dellvend Carked
Vandilia Fesigg
Regineing leage
Jauge
Vanilla
Aial (pen lonring
Bestdr
Woed Sratttutte What type of flor des the sccolees have?
Sennens Rephraing The Michen epu spedi with wnst and of "lonnng?
lndta hime!
Vadilla Quantica
Hardwasdd
Assivi::
What type
kincai obeve
relovaat
oun apple light
moder antetor There
aoso soimer Tepe What
knchte hove?
Misher hewd
boking chcllerg mhat sonedd Jil de mims, pia This csakbees fro ontorrs pu peoiie for the poctty ont onoeget tinedo When hype of floon does the leteee lowe?
WEU York Teartty
Figure 3: Illustration of our proposed dynamic multimodal evaluation framework, Vision Vision- Language Bootstrapping (VLB). (a) demonstrates how we derive insights from real user interactions with LVL.Ms, where users possess different visual attention and language understanding from diverse identities. (b) highlights the role of VLB's judge module in ensuring that generated images and questions maintain consistent with the original. (c) provides an example of VLB transforming a sample through image and language bootstrapping. Additionally, VLB can generate new, increas- ingly complex samples through bootstrapping composition.
and A denote the image, question, and ground truth answer, a dynamic sample can be represented F (V(T), L(Q), A). Here dynamic evaluation samples, respectively. V and L are transformations applied to the image and question. By comparing E,, with Ea, we see that DME is implemented by image and language bootstrapping while preserving the consistency with the original sample.
the subscripts 's' and 'd' denote the original static and
Challenges. There are three major challenges in achieving dynamic multimodal evaluation. First, the DME framework should be generic enough to be used in various multimodal tasks instead of limited to a specific task like (Fan et al., 2023). Second, it is hard to alter images and questions while not affecting the correctness of the reference answer. In linguistics, GPT4 can be used as a powerful agent to rewrite questions while preserving the consistency with the original question by following instructions (Zhu al.) (2024a). However, there is agent to achieve this in image transformation. Third, the DME framework should be flexible enough control the complexity of newly generated evaluation samples, enabling the evaluation to co-evolve with LVLM development.
Our Approach. We propose a novel DME protocol, called vision-language bootstrapping (VLB). VLB consists of multimodal bootstrapping module to generate new images and questions, and judge module to maintain consistency with the original sample. As illustrated in Figure 3 (a), by simulating real LVLM's user interaction in visual attention and linguistic understanding,
design image (i.e,, V E (V1, V2,- ...). }) and language (i.e., L E (L1. ... }) bootstrapping strategies. Experiments show that the composition of V and L would yield dynamic evaluation samples with flexible complexity, an example is exhibited in Figure (c).
4.2 IMAGE BOOTSTRAPPING
Our VLB employs image bootstrapping to build visual dynamies. Since different users pose var- ious identities and backgrounds, their levels of visual attention also vary. For instance, children's 's attention might be more easily distracted by irrelevant objects in images than educated adults. Thus, simulate disturbing focusing visual attention for image bootstrapping: adding objects, removing existing objects, and expanding original images, as Figure Each strategy is associated with difficulty level (i.e. 'Easy' or 'Hard').
Vi= Adding New Objects (Hard). Given the original sample E, = (I) V, aims to insert a new object into the image while maintaining that the answer A is still correct for generated image
5

---

Published as a conference paper at ICLR 2025
Image Fidit
Object addition
V
Add Instruttonn
Indox for Remove
Masks for Remrve
+lns
Inail 6 (plce
Object removal
SAM
Rensove Destruction.
Image Outpainting
Va
Outpanzing Batto (1.5%, 1.5x, 1,70 201
1:25
1.5t
Figure 4: Image bootstrapping strategies: Starting from an original image, route V1, V2. V/3 repre- sents the process of adding new objects, removing existing objects, and expanding original images.
(T). To this end, we integrate GPT-4V (Achiam al., 2023 and the advanced image editing model PowerPaint (Zhuang et al. 2023) in two steps. First, GPT-4V is prompted with a carefully designed add-instruction (detailed in Appendix A..1] to return a set of insertable objects. Each object is in the form of (object name: bounding box). Second, we transformed the bounding box into mask, which together with the object name and original image 7, are fed into the PowerPaint to generate the new image Vi (7). Adding objects into an image introduces irrelevant visual content, which would distract visual attention and may be harder than the vanilla sample Es.
Va: Removing Existing Objects (Easy). Simplifying the visual content by removing objects poses a substantial challenge because GPT-4V is relatively weak in fine-grained visual grounding. To tackle this problem. extend the idea of SoM (Yang al. 2023) that unleashes the visual ground- ing capabilities of GPT-4V by providing semantic marks. Leveraging SAM (Kirillov et al. 2023), obtain semantic masks M of all objects within an image. We then assign each object with serial number. Afterward, we prompt GPT-4V with remove-instruction to give the removable ob- jects, each formatted (object name, serial number). Finally, the image object name, and mask corresponding to the serial number are fed into PowerPaint (Zhuang et al. (023) to obtain the new image. Since removing objects from the image would omit superfluous visual content simplifying visual attention, Eg would be easier than the vanilla E, under 1/2.
V3: Expanding Original Images (Hard). V/3 employs image outpainting to extend the background of the image while not affecting the core elements within the image relevant to the question Q. Since expanding the image also introduces irrelevant visual content to the problem, Fa would be more challenging than the vanilla E, under V/. We set the extension ratio r = 1.5 for the main experiments. Intuitively, varying the outpainting ratio r mimics the effect of observing objects from different distances. We provide an ablation study on the effect of outpainting ratios in Section 5.4
4.3 LANGUAGE BOOTSTRAPPING
Besides visual modality, language understanding is also crucial for LVLMs. We construct language bootstrapping by stimulating different linguistic expressions of users with various identities and backgrounds. Four strategies from different levels of linguistic expressions: word level, sentence level, context level, are employed. Figure 3] illustrates the transformed samples of four paradigms L2, L3, Ca based on the original question Q. We also assigned difficulty level with "Easy" or 'Hard' for each language bootstrapping strategy.
Li: Word Substitution (Hard). Given that each user of LVLMSs possesses unique vocabulary preferences and habits, L1 simulate these variations by replacing words with synonyms or contextu- ally similar expressions while preserving the core concept for question Q by following (Zhu et al., 2024b). LLM pre-training usually has limited coverage of phrasing styles in instructions. Therefore, 21 would increase the difficulty of the original problem.
L2: Sentence Rephrasing (Hard). Considering users with different identities might express the same question with various phrasing styles. For example, the question "How does photosynthesis work?" would become "How do plants make food from sunlight?" for a casual user, while turning into "What are the key differences between the light-dependent and light-independent reactions in

---

Published as a conference paper at ICLR 2025
photosynthesis?" for researcher. To this end, Le employs role-playing (Wang et al. 2023 to rephrase the original question Q into new ones.
C3: Adding Relevant Context (Easy). L3 aims introduce visual-related text to the original ques- tion Q. The generated context should contain more hints to help understand the image. However, should be ensured that the original question cannot be solved directly with the added context. The strategy reflects the situation when users understand the image partly. We employ GPT-4V to gener- descriptions of the image via our carefully designed prompts (Appendix [A.Ip. Then we add the generated description to Q, obtaining the new question LQ()). By design, the dynamic variant Ea would be easier than the original E..
C1: Adding Irrelevant Context (Hard). Unlike Ls. L1 will supplement the question Q with visual-related text which cannot help answer the question. Here we also use GPT-4V to create context. Since the irrelevant context would distract LVLMs, the new sample Eg would become harder than the original E.,
4.4 JUDGE MODULE
DME should create new samples while maintaining essential concepts of the original one. Although our VLB employs strict criteria in the bootstrapping process, the transformation may still break the consistency. For instance, the added objects in Li might shelter key visual content relevant to the problem. To tackle this issue, we incorporate judge module to ensure consistency between the dynamic variant and the original sample.
Specifically in Figure 3(b), given the static sample E, = (, and the corresponding dynamic variant E = (V ((Q), A). the judge module is informed what modification conducted by V(T) and (()), and is asked to check whether A is still correct for the newly generated image and question. Here we designed different judge prompts to check each bootstrapping strategy, detailed prompts are in Appendix A.2 Similar to MPA (Zhu al. 2024a), the judge operates in an adversar- ial manner and returns a Yes' or 'No' verdict. If the response is 'No', the sample will be regenerated until it passes the assessment. If the sample does not pass after five attempts, the original sample is used instead. In practice, we use internVL-2. an open-source powerful LVLM (Chen et al.] [023] 2024b), capable and affordable judge. What's more, further validate the effectiveness of our judge module by the human verification in Section A.5
4.5 COMPOSITIONAL BOOTSTRAPPING
Due to every single VLB strategy for image and question being atomic, we can investigate two kinds of bootstrapping composition with flexible complexities. 1) Paired multimodal composition. We can compose visual bootstrapping V and linguistic bootstrapping C, into a paired multimodal dynamic sample Es (V) (Q), A). obtaining total of 12 dynamic variants. 2) Multi- strategy composition. We can also stack multiple image bootstrapping strategies on a single image multiple language bootstrapping strategies on the question, composing multi-strategy dynamic sample like E. = (V) L))
Since each single VLB strategy possesses different levels of complexity, the above two kinds of compositions can effectively construct different variants varying in complexity, to assess the robust- ness and adaptability of LVLMSs and explore models' upper and lower limits in performance across different benchmarks. The details and results of the strategy compositions are in Section 5.3
5 EXPERIMENT
5.1 EXPERIMENT SETUP
Tasks and Datasets. We selected five popular benchmarks to assess current LVLMs, encompassing Yes/No Questions (MME), Multiple Choice Questions (MMBench, SEEDBench), and Visual Ques- tion Answering (MMvet, LLaVhhenchh. These benchmarks include broad spectrum of cognitive and comprehension tasks. In Section 5.2 and 5.3 we employ three comparable datasets in terms of size: MME, MMBench (30%), and SEED (10%) as the experimental datasets. Then we extend our dynamic strategies the full set of MMBench, MMvet, and L.L. VA in Section 5.3
7

---

Published as a conference paper at ICLR 2025
Table 1: Results of three image bootstrapping strategies on LVL.s. For each benchmark, 'Vanilla' represents the original results, while V1, V2, and V3 represent the results after image bootstrapping. (The numbers parentheses show the difference compared to vanilla, with red indicating decreas- ing, and green increasing.)
Modkl
SIME
Vavilla
Fi
173
Viniila
V1
1/2
33
Veriilo
3/1
3/3
20:41
63
70.39 01
10.15.3
13.90(13:00_)
71.86-1 7011
89.00
(0:93")
2013
73:85-0333
15.37 (4.27
79.72.00051)
76.89-2.7533
86.15 (2.00
71.54
(5:54.)
11.88
(3.1)
75.26
14.04
17 5011
65.46-0.511
VL.Cher
(1:54.)
1142
08.04
67.31
17:57-0:551)
77.69 851
74.33
2013
78 (3.799
XOmpper2
15:40
00.18 (6.22.)
05.19:0.2113
11.13 14.27
$4622
17.23 (139)
(1577))
76,89 (2.75)
92.32
09.20
(3.11
99.15 (0937)
08.15
07.68 [5-41_3
67.88 966)
05.34 (3.01.)
8:61
18.09 (6.31))
31 (1.60.117
00.00
31.25
(7.6)) M111
76.516035D 17776117777
16.40
70.16 (5-3
37407 (1777)
28.45 (2.34
$8.59
81.13 (1-36)
89.28 (0-561)
83.01 01 5711
10.01
72.20-22511
92-11 12:2979
20-522 28-503/90)
GPT-4
77.39
71.9
(668.))
78.18 (0.52))
73.70 (3.95)
$7.37
82.18 (5,68])
29.06 (1.65))
85.28
(2.09))
77.57
71,04
(6.523)
78.49 (D021))
(5.90
Table 2: Results of vanilla and applying language bootstrapping strategy L1, L2, Ls, L4 on LVLMs
www.Bbookk 201
Chellall (91
50a 200
Vaali
C4
Todda
6945
08.11 00-110
50
10 0030050011
30.02
18,1111-000
18:00 31RU
29.30 00.34133413
29.9
350010
01.01
200 1017
21.000
18.49-1 9012
63.22
0800 01133
00.44 (358
4146-2 2-511
43.9 186643
11.64
6829
08.33 1331
09d612-301
14.36
63.21 85.3
54.8 02 18U
08.35
68.44
4..01155
11,903 541
81.05
8-5 X4870
8639147313
28.8 12471
80
7646 213
0921 (3.00)
05.80 (1577
Office
17.99
17460
75 (1173
0799/2991)
71.97
76 341
3199 0L.
74.0594441
73.994 2013
Evaluated LVLMs. Our evaluated LVLMs include both closed-source APIs: GPT-4o (Ope- nAl 2024) and Claude (Anthropic) 2024). and open-source models: DeepSeck-VL. (Lu et al. 2024b), TransCore M (Research) 2023)) XComposer2 (Dong et al. 2024), Monkey-Chat (Li et al., 2024b), LLaVA-NeXT-Vieuss (Liu et 2024a). Yi-VL-34B (Young et al., 2024). Qwen- VL. (Bai et al. (023) and IL (Chen et al. 2023). We utilize the standardized evaluation platform VLMEvalkit (Duan et al. 2024) and set the generation temperature as 0 for all evaluated LVLMs to ensure fair comparison.
5.2 RESULT OF SINGLE BOOTSTRAPPING STRATEGY
The results of image bootstrapping strategies. We apply our designed image bootstrapping strate- gies: Adding New Objects (V1). Remove Existing Objects (V2), and Expanding Original Images (V33) on existing benchmarks. Table T presents the evaluation results across series of LVLMs. Con- sistent with predefined difficulty levels, V1 and V/3 generally result in a decrease in accuracy, and V/2 slightly boost the performance. It indicates that adding irrelevant objects or expanding the original image can introduce more superfluous visual content, thereby hindering LVLMs from capturing the key visual content related to the problem. Meanwhile, removing some irrelevant, extraneous ob- jects can simplify the visual information of the image, allowing LVLMs to better focus on the key elements. More visualized cases are in Appendix A.3 A.4
The results of language bootstrapping strategies. We also apply linguistic bootstrapping on different linguistic levels, including word substitution (C1), sentence rephrasing (C2), relevant context([3) C3), and irrelevant context (C4). Table 2 illustrates that, except for L3. the other strate- gies result in a degradation of LVLM performance. This highlights the challenge LVLMs face in addressing questions posed by different individuals from diverse backgrounds. Therefore, further efforts are needed to enhance LVLMs' ability to comprehend the semantic essence of questions and improve their user adaptability. These findings align with our previous revisiting experiments, suggesting that benchmarks may be contaminated. Conversely, incorporating relevant captions C3 helps the model better understand images, boosting accuracy. Quantitatively, employing C3 and C4 presents more fluctuation than Ll and L2. This may be due the situation (Kamradt 2024] that longer text poses greater influence for LVLMs compared to shorter words and sentence.
5.3 RESULT OF COMPOSITIONAL BOOTSTRAPPING STRATEGY
The performance of paired multimodal composition. We combine our strategies on and L and obtained 12 dynamic variants that integrate visual and linguistic bootstrapping. Figure shows the composition results in MMBench across three LVLMs, where each axis is organized from easy to difficult based the complexity levels obtained from the above single-strategy experiments. We
8

---

Published as a conference paper at ICLR 2025
LLWWN NATT
Intemit-Chaa-V155
YLVI
Accurucy
Inagge Breirapiiggg
92.45
73.47
72.53
82.05
32
$0.38
79.97
Imagee Bredmoppig
V1
$4.37
7641
73.53
72.4
05:32
75.21
2121
70:35
81:71
92.51
VP
$5.93
78.
7722
22
12
$2:47
76.8
78:48
7718
88.06
60.4
64M2
E4:8
V2
66.07
78.0
79.66
78
12
L4
LB
L1
12
L4
Language Bosstrapping
Language Roostraphing
Langunge Besctrspping
Figure 5: Results of composing image and language bootstrapping strategies MMBench.
Table 3: Results of the hardest and easiest variants after multimodal bootstrapping strategy compo-
sition. The 'vanilla" represents the original results, while the hardest is V + L4, easiest 1/2 L3.
Model
MMBeach DEV Pall (5)
MMvet
LLaVARench
(V1-C4)
vanilla
Fasy (V2443)
Hand (V1+54)
vanilla
Easy (V2+C3)
Hand (V1+C4)
vanilla
Easy (V24CC3
DeepScck-VL
73.06
40
LLaVA-NeXT-Vin Vicuua
71.00
76.41
82.83
24.22
47.56
63.02
15.
69.00
XCoepossr2
76.94
$3.02
$7.08
30.55
18.45
60.55
42.30
58.00
73.50
YL-VI-
71.40
77.91
85.30
22.15
10.55
60.23
37. 50
44.30
59.70
Vn-emVL-2
78.00
$6.78
88.17
36.35
51.19
68.15
36.00
53.00
68.30
GPT-Mo
79.55
85.27
86.32
43.66
73.94
77.15
55.20
76.00
78,80
69.04
82
85.17
26.10
70.77
75.91
36.30
70.50
75.8
observe that these LVLMs exhibit varying performance when faced with different variants, with overall trends being relatively consistent (i.e., performance gradually decreases from the lower left to the upper right corner). Specifically, the combination of V1 + L4 presents hardest challenges for LVLMs, while 1/2 + C3 significantly boosts performance. These findings align with our obser- vations from applying single-modal strategies. Results of paired multimodal composition more benchmarks and LVL.M can be seen in Appendix figure
Based on the above result, extend these extreme compositions (i.e., V1 L4 and V2 L3 ) to the full set of MMBench, MMvet, and LLaVABench, to demonstrate the universality of our dynamic evaluation strategies. As displayed in Table 3 our VLB could be effective in evaluating the upper and lower bounds for the models' performance. This capability to transform an original sample into versions with varying complexity not only tests but also enhances our understanding of the comprehensive capabilities of LVLMs. Through VLB, we achieve a more thorough evaluation, highlighting the robustness and adaptability of LVLMS to handle diverse and complex multimodal interactions.
The performance of multi-strategy composition. Due to every single VLB strategy for image and question being atomic, we can explore the effect of different compositions according to the number of applied strategy: vanilla, single-strategy (V1), dual-strategyy()) +V3)) and tri-strategy (x+))++C,). Together with the easiest composition (V2+Lz),, Figure 6 demonstrates the perfor- mance of three advanced LVLMs on SEEDBench and MMvet. The lines show that as the number increases from 0-3, the difficulty of the benchmark also increases, leading to gradual decline in the accuracy of L.L.M.
of hard strategies
5.4 ABLATION STUDY AND ANALYSIS
The effect of outpainting ratio in V3- Since different expanding ratios can result in varied visual content, we investigated the influence of different expanding ratios (1.25x, 1.50x, 1.75x, and 2x in our V3 strategy. The results in Figure [ indicate that as the expanding ratios increase, the accuracy of the models in correctly answering questions decreases, and the rate of this decrease becomes steeper. This can attributed to the fact that a larger expanding ratio introduces a more prominent and possibly distracting background into the image, whose visual information may not be directly relevant to the question, thereby diverting the LVLM's attention from the crucial cues.
Can our VLB reduce data contamination? Our dynamic strategy mitigates data contamination in two ways. Firstly, since the new images and texts are generated using image editing tools and GPT-4V, where the process is random and variable, it is difficult for models take the vast array of random variants into training. Secondly, the newly generated sample differs from the contaminated original sample, thereby reducing the similarity with training data. We applied the same methods

---

Published as a conference paper at ICLR 2025
redetect the hardest variant in Figure 8 and found significant reduction in data contamination rate among training sets.
of Section B
80
90
88,55
SeedBench
85,17
Accuracy (%)
60
Accuracy
73,36
78,48
77,55
76,23
30
70,52
69.95 59
Claude3-SV-Sonee
67,66
InternVi.2
54,48
10
0.00
1.25
1.50
1.75
2.00
Outpainting
Figure 6:
The accuracy of LVLMs to-
wards different multi-strategy compositions Figure 7: The average accuracy of all evaluated on SEEDBench.
LVLMs different ratios of outpainting scales.
The effect of VLB on different tasks. Since our dynamic strategies are applicable to a wide range of multimodal tasks, utilize SEEDBench demonstrate the performance changes of LVLMs across various tasks. F presents a comparison of average performance among the vanilla benchmarks, and two dynamic variants with 1/2 + C3 and V1 + L4 strategies. It is evident that performance changes more significantly in tasks like 'Instance Interaction', 'Text Understanding' and 'Spatial Relation', We found that images in 'Instance Interaction' and 'Spatial Relation' contain fewer objects, making the addition of new objects particularly disruptive to the LVLMs' focus on core visual elements. For "Text Understanding' it aligns with our strategic design hypothesis that introducing noise contexts can distract the models from understanding essential texts. Detailed error cases these tasks on GPT-4o are in Appendix A.6
33.13
20.38
25.56
SEEDBench. MMBench. MME SEEDBench.s MMBench MME
100
Contamination Rate (N)
18.72
Test Underrtaanddg
Intance Intmecte
13.45
10
18:77
7.33
6:27
3.57
Coro-Caption
Image-Text
Scece chdeestatdigggg
WAlHoes Caandin)
The accuracy for the hard rate between the dynamic variant and the vanilla variant, easy variant, and the vanilla static benchmark s.
Figure 9:
Figure& 8: Reduction of image-text data contamination
static benchmark across different mul- timodal tasks on SEEDBench.
6 CONCLUSION AND DISCUSSION
In this paper, we begin by quantitatively detecting the degree of contamination present in current benchmarks, indicating the need for dynamic multimodal evaluation (DME). And then we develop the first DME protocol with flexible complexity via Vision-Language Bootstrapping (VLB). VLB introduces bootstrapping strategies designed to transform original images and questions into new versions that simulate real user interactions in LVLMs, focusing on aspects of visual attention and linguistic understanding. We also employ predefined rules and the judge ensure the newly gen- erated problem preserves the consistency with the vanilla. With VLB, we build a series of dynamic variants upon existing benchmark with diverse complexities. By conducting extensive evalua- tions using VLB, we demonstrate the limitations of current LVLMs, highlighting areas for potential improvement of LVLM in user interaction and adaption.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This paper is partially supported by the National Key R&D Program of China (N0.20222D0160100 NO. N0.2022ZD0161000 and the General Research Fund of Hong Kong 17.17200622 and 17209324. This work was done during his internship at Shanghai Artificial Intelligence Laboratory.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0860 08774,, 2023.
Anthropic. Claude 3.5 Sonnet, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are on the right way for evaluating large vision-language 2024a.
models? arXiv preprint w:3:::2443.203333
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: arXiv:/504.00 00325, 2015.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312. /4238, 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-244 24198, 2024b.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the 2021.
colossal clean crawled corpus. arXiv preprint arXiv:2
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang- intemlnnxccopooee2:: Mastering free-form text-image composition and comprehension in vasion-language large model. arXiv preprint arXiv:2401. 16420, 2024.
Haodong Duan, Jumming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vimevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL htt.pss://exxi..ogg///po
2407.11699
Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, Yongfeng Zhang, and Libby Hemphill. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complex- ity classes. arXiv preprint arXiv:2312. 14890, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.

---

Published as a conference paper at ICLR 2025
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family multi-modal large language models. arXiv preprint arXiv:2402.05995, 2024.
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-Iol: Visual question answer- ing under the lens of logic. In Europeun conference computer vision, pp. 379-396. Springer, 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1904-6913, 2017.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2004.08718, 2021.
Geoffrey Kamradt. Limtest cimeeestnecdlliinaaaattta...
https://github. www wtt.///w...oooo//.oac.caoa
LLMTest LIMTest_ NeedleennHaasttack 2024. Accessed: 2024-10-01.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Ab- dullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, et al. A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. arXiv preprint (eXX:x2407.04000 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal ilms with generative comprehension. arXiv preprint arXiv:2307.66125, 2023.
Yucheng Li, Frank Guerin, and Chenghua Lin. An open source data contamination report for large language models. arXiv preprint arXiv:2310. 003310.77588 17589, 2024a.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
ilava-vi. wit.bb.......................... ho/blog///aaoeeoooooooooooooooeo 2atpp///222220--02200000 300--300-laaaaa -1aa--11aaaaaeeeaaaaaaaa
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseck-vl: towards real-world vision-language understanding. arXiv preprint Xrviv::403.0000 05555, 2024a.
Haoyu Lu, Wen Liu, Bo Zhang. Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEEleyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
OpenAI. Hello GPT-4o, 2024.
12

---

Published as a conference paper at ICLR 2025
PCI Research. Transcore-m. https////gggggaeee.c//////aaaaaa 2023. Accessed: 2024-09-29
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
Ramprasaath Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Be- smira Nushi, and Eee Kamar. Squinting at vqa models: Introspecting vqa models with sub- questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10003-10011, 2020.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6649-6658, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 3rXIv:2302.1397,, 2023.
Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, er al. Rolellm: Benchmarking, eliciting, and enhancing pole-playin abilities of large language models. arXiv preprint arXiv:2310.00746. 2023.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang. Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.0922 02060096..0 2023.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 11441, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403. 0445222 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 23088 02490, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVE conference on computer vision and pattem recognition, pp. 6720 2019.
Kaijie Zhu, Jiaao Chen, Jindong Wang. Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Graph-informa dynamic evaluation large language models. arXiv preprint arXiv:2309. 2023.
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. In Forty-first International Conference on Machine Learning, 2024a.
Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research, 25(254):1 2024b.
13

---

Published as a conference paper at ICLR 2025
Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.00354, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 BOOTSTRAPPING STRATEGY INSTRUCTION
In this section, we detail the instructions for GPT-4 during the bootstrapping strategy process in Table They are strategy V1, V2, L3 and L1, respectively.
In strategy V1, we designed add-instruction, to prompt GPT-4V to return a set of insertable objects. Each object is in the form of (object name, bounding box).
Since GPT-4V is relatively weak in fine-grained visual grounding, in strategy V2, we utilize seman- tic masks to facilitate the appropriate selection of objects for removal. Leveraging segmentation models, we obtain semantic masks M of all objects within an image. Upon identifying semantic masks M, each object is assigned a serial number. GPT-4V requires pinoint the objects suitable for removal based on their numbers and names, each formatted as (object name, serial number).
In strategy L3, we seek to enhance the context with a caption focusing on information related to the question, generated by GPT-4V. Thus, we query GPT-4V to generate additional information that aids in responding.
In strategy L1, we aims at introducing nosing context, which pertains to adding context that is related to the image but not directly pertinent to the question. For instance, an observer might initially notice another object or the overall atmosphere before focusing on a specific object about which the question is posed.
Aspect
Inpat
Instructions
There is question about the image (resolttion!!! x W)) Tts question < Q >. Its cornect inswer choice ist < Now please add as object into this image, but keep the original answer of the question the same, which means the added object can not charge the criginal answer
and the position of the added object can not cover the visual answer informacion.
Vi(I) Image Boosstapping
Please give randomly objects be added and give the original resolution give you.
Note that the size of the added object should be larger.
bounding coordinates
each object scceeding
The box cutpui is coordinates Th ourput fonmat should only
upper comner (Xmrs, Sirre) lower right corner (Xraan Biser) the object.
Each iem in list must he diet. Do not outpui any extra information! lust ourput list. example of ouppur format
(name : name of objectl, box: [Eanns Raier Puces Mosal | name : name of objeci2, box: [Fruin Mesin Knusss Yean
Resed een the image I provided (image includes the wark and the mork for each objoct).
tell me which objects can be removed without changing the answer to the question < Please give list containing extetly 5 objects can be removed. Do not cutput estra infocmation. Just
Po(() Image Bootstrupping
E,,
jobject mark: object list. The example of oupput format is object mark: specific objectican eppeated without distinction).
object
where objjec__mnck is given Nute the removed
and objoct same
can not change unswer question.
the order you think most obvicus appropriate ressove.
Cyll) Text Bootstrapgngg
<1Q
Please ald comext the question introducing context or details the questio that are relevant bui not directly helpful in answering it. Refer to the acoompacynng image < 7 for contest. Question: < Q >.
Now, bised en this image, I want you to provid paragoph than inclovant to the question: Q >>. The paragraph dbould not help in answering the question x all. Be careful not directly descriee the image, as that would add answering the question. Instood, you should focus unrelated object element the image I >., and your imagination wild. The creative ofT-topic,
(,( Text Bootstrppping
<1.0>
Plense the paragraph should
than 100 words
Table 4: The detailed instructions for image and language bootstrapping strategies.
A.2 JUDGE INSTRUCTION
Although our dynamic pipeline adheres to strict criteria to maintain the consistency of the original intent and informational content, it still unintentionally alters the meaning. Therefore, we intro- duce the judge module to compare the original with the modified images and the original with the rephrased questions, ensuring the preservation of the essence and factual accuracy. Specifically, to minimize excessive perturbations, the judge module assesses the newly generated images and ques- tions from both the visual and textual modalities, respectively. Details of instructions can be seen in the following Table 5
A.3 MORE CASES OF IMAGE BOOTSTRAPPING
We provide some cases transformed by our image bootstrapping strategics in Figure 10 Visually, these dynamic variants show consistency with original images, ensuring the preservation of the
15

---

Published as a conference paper at ICLR 2025
Ve will no megine oen anthe te apn whe celiener dhaign- weddaaaaa taae
<,00000>>
aid states waaeeo
(rugel) Jemeerea saa You
Rerpeddant praae and
straps odet coniineriiieen
on fertee
ToY Ne"
statc Bi decegial laus thags dester Yor oo eay.
Teuge T
Pe
teage Mymic
The Ricoiiiiiie bilone 2 with anocust Muaee carti coqur beag
Ter Pg
Bodt
Gooal Trak Phnteppon goiken
Oe Oumdadd (.00)
Nor7
<R.440)>
Pizas wheter
peneers (310)
Bagg This aquntir out repiday
Talied dsciiaam
Ter he"
Vayy "oo and
Only
<
ce te desudioo dee gndie Blrog
aud Newey bay buduuder -Q's
Table 5: The detailed instruction given to the judge module.
essence. However, for LVLMs, the scenario is quite different. Despite the visual similarity, these modified images present new challenges.
Object
Remove Object
Expan ding image
SEEDBench
Queetion bot type flootnng does have?
MMBench
Quaston What typ envinnmentts Caplcaed pictum??
MME
Addial
Questonis them baseball botin this this: Ploss oro
Figure 10: More cases of image bootstrapping on SEEDBench, MMBench, and MME
A.4
MORE CASES OF LANGUAGE BOOTSTRAPPING
We present some results of bootstrapping the textual modality of existing benchmarks in Table 6
A.5 THE EFFECTIVENESS OF JUDGE MODULE.
Although our newly transformed images and questions have passed adversarial judged, for the sake of rigor, we further conduct human verification to ensure that our variants maintain consistent with the original benchmark both visually and linguistically. For each dynamic strategy, we recruit 20 human experts (with bachelor or higher degree) and randomly selected 100 samples from each strategy for each benchmark, totally 2100 samples. They are also tasked with judging the integrity and appropriateness of the modifications. For visual dynamic strategy, we compose the original and edited image, with the corresponding question and answer, into the questionnaire. The experts required to verify: (1) Whether the edited image maintains the same answer the original image. For linguistics dynamic strategy, we compose the original and transformed question, with the corresponding image and answer, into the questionnaire. And ask (2) whether the original and paraphrased questions were equivalent. The evaluation required a simple 'Yes' or 'No' answer for asked questions (1) and (2).
Experts independently reviewed samples. Once all experts completed their evaluations, a voting process ensued. A sample was only considered to have passed if more than half of the experts
16

---

Published as a conference paper at ICLR 2025
Benchourk
Stratce
Questio
Onginal
Whst Besery dop kischen hovel
What upe Dooring does costromn
BEED
The kikhee eouipsl
thowing?
6x10)
Asblis and medem kikiccs iitetir desan, There a lage wisdow abore lhe sest that allors auple saland lale to Boed spee
The countcctops appeur aodid surliss, poadly narlic qusts
ope Dootin
kiichen
Us the kitchen cxpteard, there cookbook
lo baking ctailenge uprzads de sove
4,90
The cooltbook traneiome as everday lotchen into a aeses communiy, whaie the atisi dhece of fos and sagar iyrace passon for the paity ars amsegge frieed.
What type of lisery don liiches
What of enrnomment is depicaed potres?
4,00)
What ope of sameedding iedpppeed picture?
What
de
The image ponmiy a midim. minirelia liiing quon fonrring open living and dining sera.
The norn lecoqponoe sratna polets
cartly toos aeople rumural light
prsurace enard has hncepphon brings nush roter indocn, emplocoiin mrch to cony The Proemmmeeenti is cartiilyy orgaline and dreggal io emre hurmerini and tangul living spece Goon the denils mmegy. the depicaad mnniemer om had dranthel moders mistmable, and cory indker living spoce typ
smmopphern
depicadd
Insgire Frieczed ineearii moterr al fumiture was koorgiaeerd, possesing life cyle akin to plants or asimals
Cossider soft thut guiets small cedtiiny dooms eoch qring. wikkh over serreer gradeally booo into leds velrety sarficn per for lounging. snturnn approchees befon pivdl shooling
traesfonn
spectacitte diglly
polss, and oranges.
distmilaa decifoons
The cycle werld thn culiiizaic persod of diemaayy duesgg wists, when the aquahes secgglt casure undaal ins couily sosplendence
whon wa assi retuns. What type civanaracci is deyectad paturt
Ungesal
In dune hew het thi fgua Plavo snowe or
0,10)
Dues Dore ckis
image?
gire tee snswer just you Bo.
The image depico seme fiom luacial
The bater stesiing is dee bulles bus,
6x40)
axspire stands belind calchot slecning lhe sone Joaels The degoot is backgooadd is popuiaee will lesu undees wakhing the saree
poised snics, while the watkee a cosaced bekind the lone place coals to cash the bull.
imgee
In the world of spotting suir the evaation of the calceers mit panseslarty Eacisaing.
piootal piece equpmen, cualled mallicen cid the pame tecll undeppon consukmmti easdiemaiio year. sillnd
4.99)
daaall mavellss wihhtadd oppattee Nighirppar cartee
Thes mins ner only prenett the plyer's hond hut aloo echonce fhir abilyy to bocorc Eut modlnn bolls with przckinn thoocwing porfect Ward bettage and irmonaico sport trcheelogy
image?
you
Table 6: More cases of language bootstrapping on SEEDBench, MMBench, and MME.
deemed it consistent with the original. In cases where the voting resulted in tic, the sample was regarded as ambiguous and deemed not to maintain consistency with the original. The positive results are shown in Figure 11 our human verification obtain a high accuracy rate, average of 96% for image dynamic strategy and 97% for question dynamic strategy on all three datasets. The result showcases high level of alignment between the judge model and human verification, proving the equivalence and correctness of our dynamic methodology.
Ses0Bench Bll Milllench MME
41
Figure 11: Human verification rate on each bootstrapping variant maintaining consistency with the vanilla.
32/00ench
MV3mnch
NNT
Figure 12: Results of composition between image bootstrapping V and language bootstrapping C strategy.
17

---

Published as a conference paper at ICLR 2025
A.6 ANALYSIS OF ERROR CASE
We visualize the GPT4o's responses to images before and after dynamic changes. Figure 13 a) shows an example of instance identity task, where adding a sofa caused the LVLM to wrongly answer question it correctly responded in the original. This may be due to the added red sofa, which is brightly colored and distracts the model's attention. Figure [3] (b) is an example of spatial relation task, where removal of cabinet made the relative positions the fan and stage clearer, allowing the LVLM to correctly answer the question it had previously gotten wrong.
Instanee Interaction
Spotial Rdlation
Quastion the girl weaning any occessoniee les?.
Questiors
stago?,
bal wooning cny accessciass
from the stagn
You ahe veaming her carnaga
Acavt the stope. eliee
Figure 13: (a) After the Vi strategy, GPT-4 incorrectly answered an instance identification question it had previously answered correctly. (b) By V/2 strategy, GPT-4 correctly answered a spatial relation question it had previously answered incorrectly.
especially about questions before and after dynamic changes. As be seen, our dynamic strategy for generating multi-modal VQA pairs indeed results in performance changes, presenting a greater challenge for LVLMs.
We further visualize more of LVLMs' responses in Figure 14
Question: Based smount fortorbbbly this livin room?
Origini question What's demonetrated object?
4-5 people
demerstrated use?
Origital A
NarCholce B
Origiaal Chalee
New Cludas
Cornect onswer C
60 Vi-Mdi
(b) 1.l-Repprradggggnnnnne
questio on guin demeanar of porrets quite spachacle trepical andinssments, vibrast hinds joy
Or gindl question In this comperison picture, are madide right
longthy phroses. luster reflects stuned glass windows on pawny day. The parnet's diet equaly calarful, consisting priteslly Iralts, ruts, and seeds, which They adepiily pick and cheece with their streng. curved books. Obsersing thede whinesic creattunee es their o delginful escape wonders
bird Their foothers
unconey
peniniscent
Question comporisur picture.
Ociginal Chaice:A A
Zome
mall
joigo
63 1-4Askl invelvnnt contost
Figure 14: More етrоr in LVLMs and more strategies. (a) After the V1 strategy, GPT-4 incorrectly answered a question it had previously answered correctly. (b) By L2 strategy, InternVL. incorrectly answered a rephrased question had previously answered correctly. (c) By Ca strategy, deepseck-vl incorrectly answered a question with irrelevant context.
A.7 EXPERIMENTAL DETAIL.
A.7.1
THE SPECIFIC VLMS PARAM VERSION
Table
lists the specific model names and their parameter version evaluated in our experiment.
Model
DeL
TransCorc-M
Monkey-Chat
LLaVA-NoXT-Vicuss
Qwen- VL-Char
XComposer2
Yi-LL
IntemVL
Size
7B
28B
9.8B
13B
20B
7B
34B
8B
Table 7: Comparison of model sizes across different architectures.
18

---

Published as a conference paper at ICLR 2025
A.7.2 EXPERIMENT ON THE PERFORMANCE VARIANCE
In our practical experiment, for each image and language strategy. we set up five random seeds, gen- erating five data variants. Therefore, all experimental results are the average of these five variants. Below we provide the standard deviation for each strategy across the five variants.
Dataset
Model
VI
V2
V3
L1
12
L3
L4
TransCrre-
69, 10 (0.575)
73.86 (0.5026)
69.52 (1.3760)
72.54 (0.0353)
71.67 (00330))
71.95 (0.0599)
72.19 (0.0319)
SEEDBench
DeepSeck
(0.7716)
70.35 (0.1560)
(0.0523)
68.51 (0.0175)
(0.0655)
70.16
InterVLL2
70.46 (0.6834)
77.67 (0.6368)
74,45 (0.5221)
75.26 (0.0486)
74.77 (0.0487)
77.24 (0.0630)
74.31 (0.0444)
TransOoreM
75.37 (0.6665)
79.72 (0.3891)
76,89 (0. (022))
78.53 (0.0056)
78.62 (0.0032)
84.89 (0.0050)
78.55 (0.0016)
MMBench
DeepSeck
75.90 (0.27)80)
80.03 (0:2573)
77.96 (0 (0.2527)
78. 15 (0.0037)
78.92 (0,0025)
84.26 (0.0042)
79.07 (0.0044)
InterVI1.2
(1.0671)
(0.7612)
83.01 (0.2827)
(0.0032)
(0.0015)
(0.0058)
85.53 (0.0049)
TransCore-
(1.1170)
(0. 1922)
(0.6643)
(0.0298)
86.31 (0.0522)
(0.0534)
72.92 (0.0699)
MME
DeepSeekk
78. (1 2455)
86.59 (0.0841)
82.69 (0.6797)
85.67 (0.0425)
84.05 (0.0181)
92.23 (0.0478)
71.18 (0.0003)
InterVLL VL2
72,70 (0.4262)
82.31 (0.7943)
76.54 (0.9789)
77.15 (0.0277)
79.34 (0.0474)
79.52 (0.0563)
69.98 (0.0455)
Table 8: Mean and standard deviation (in parentheses) of accuracy across different strategies and datasets.
the standard deviation in each strategy slight, thus the variance scale caused by the randomness of GPT-4V and PowerPaint is also minimal. These demonstrate the reliability of the average accuracy result reported in our paper.
As can be seen from Table 8
A.8 ABLATION STUDY OF IMAGE EDITING MODEL
To explore our framework on other models with similar functions, we selected two other popular image editing models to replace PowerPaint. For strategy v1, we utilize BrushNe[] for the object addition step and keep all other steps exactly the same with PowerPaint. For strategy V3, we apply Stable Diffusior] for image outpainting with the same ratio x.5x. The table below shows the eval- uation results of our framework with PowerPaint, BrushNet, and Stable Diffusion on SEEDBench across three LVLMs. It is worth noting that we also generated five variants for each setting and take their average results.
VI
V3
Model
vanilla
+PowerPaint (L)
+Brushnet (J)
vanilla
+Pooerpaint (3)
+SD outpainting (U)
TransCore-M
73.58
69.10 (4,48))
70.91 (2.67.)
73.58
69.52 (4.061)
70.45 (3.13J)
DeepSeek
69.44
64.79 (4.65L)
66.26 (3.18.)
69.44
69.39 (0.05L)
69.14 (0.30L)
InterVL--2
76.80
70.46 (6.341)
71.36 (5,44.)
76.80
74.45 (2.341)
75.05 (1.75L)
Table 9: Accuracy comparison of VLB under different image editing models for VI and V3.
As can be seen from Table D although there are minor numerical differences between the results from PowerPaint and other editing models, their trends are similar. Therefore, the experimental conclusions in our original paper are valid and reliable. What's more, since our framework is verified to be adaptable to many tool models with similar functions, then our framework will perform better and better as these tool models are improving.
A.9 ABLATION STUDY OF JUDGE MODULE
We further present the results of human verification on consistency before and after the judge mod- ule. In specific, we sampled data from generated images and questions that do not undergo adver- sarial selection by the judge module, and conducted the same human verification as A.5
As shown in Table TO it is evident that for most strategies, the verification rate is significantly improved after the judge module, demonstrating that our judge module is an effective automated verification technique. The increased approval rate for the visual strategy demonstrates the effective
Ju, Xuan, et al. "Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffu- sion." arxiv preprint ar Xxxx:2403300697 (2024).
Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the TEFF/CVF conference on computer vision and pattern recognition. 2022.
19

---

Published as a conference paper at ICLR 2025
Dataset
V1
V2
V33
L1
L2
L3
L4
SEEDBench
83
92
91
89
93
93
98
Before Judge
MMBench
81
94
88
91
90
94
95
MME
84
95
87
87
85
95
97
Table 10: Human verification rate before judgment on different benchmarks and strategies.
judgment of our judge module in image-text alignment, while the improved approval rate for the linguistic strategy reflects the effective verification terms of semantics.
Dataset
V3
L1
L.2
SEEDBench
0.3477
0.1014
0.2441
0.3428
0,0307
0.0303
0.0084
MMBench
0.4395
0.0906
0.2729
0.3092
0.0158
0.0867
0.0567
MME
0.4019
0.0973
0.2063
0.2085
0,3263
0.0548
0.1863
Table 11: The average adversarial iterations by our judge module for each strategy.
We have also provided the average adversarial iterations by our judge module for each strategy in Table 11 The iteration number means the average times of samples be rejected and re-selected do not pass) by the judge model. This demonstrates that our judge module is an effective and intelligent filter, which can select accurately modified samples that are consistent with the original.
A.10 MORE ANALYSIS EVALUATION RESULT
Visually, we analyzed the relationship between the ClipScore difference value and the accuracy dif- ference value. For strategy VI, V2 and V3, we calculated the CLIPPScoe between newly generated images and the original images, together with the average accuracy difference of each strategy. Be- low we take MMbench as an example for analysis. As shown in the Table [2] it can be generally found that, images with lower CLiPScore with the original image cause greater accuracy changes for LVLMs. This indicates that the relatively bigger modifications on the original image can lead to more significant visual attention disruptions and thus pose bigger challenges to the LVLMs.
strategy
V1
V3
V2
Accuracy Difference
5.7655
2.2777
0.4133
CL.IPScore
0.8150
0.8776
0.9502
Table 12:
Relationship of accuracy difference and CLIPScore on MMBench. The table ordered by the accuracy difference value compared the original results for each strategy.
strategy
L4
L3
L1
L2
Accuracy Difference
3,2211
2,900
1.5944
1.1433 9.3083
Question Length
85.7337
63.6534
8.8523
Table 13: Relationship of accuracy difference and Question Length MMBench.
Linguistically, for strategies L1, L2, L3, and L4, we calculated the word number of questions, namely question length. Then analyzed the relationship between the question length and the accuracy differences, as depicted the following Table 13. Generally, it can be observed that the longer question can lead to more accuracy changes for LVL.S. This also aligns with the conclusions in previous work] which reveals longer contents pose greater challenges to LVLMS. Therefore, the result of our linguistic strategy on LVLM reasonable and systematic.
A.11 DETAILED EXPLANATION OF USER INTERACTION
The user interaction with LVLM is defined by the cognitive process through which users comprehend the image and ask questions of their interest in the VQA setting. This process is influenced by visual attention and linguistic understanding.
Wang, Weiyun, al. "Needle In A Multimodal Haystack." " arXiv preprint arXiv:2406.0721 (2024).
20

---

Published as a conference paper at ICLR 2025
Specifically, Visual attention affects the way how individuals selectively focus on specific visual elements within an image or scene while filtering out less relevant information. Since different users pose various identities and backgrounds, their levels of visual attentior] also vary. For instance, children's attention might be more easily distracted by irrelevant objects in images than educated adults. We categorize visual attention with concentration and distraction. The attention concen- tration is implemented by object removal while distraction is implemented by object addition and image outpainting.
Linguistic understanding indicates how individuals interpret and comprehend questions. This pro- cess influences the language proficiency of different users. Specifically, when different users utilize LVLMs, they may have different linguistic expressions towards a same question due to their distinct identities or educational backgrounds. Therefore, as shown in Figure our work conducts trans- formation on the original question from three linguistic leveld] word-level (L1), sentence-level (I.2), and context-level (L.3, L4)..
(1) Visually, as described in section 4.2 since different users pose various identities and back- grounds, their levels of visual attention also vary. For instance, children's attention might be more easily distracted by irrelevant objects in images than educated adults. Therefore, shown in Figure 3(a), we simulate the distraction, simplification, and expansion on visual attention for image boot- strapping. respectively corresponding to our three strategies adding new objects(V1), removing existing objectssV2), and expanding original iV3) Specifically in figure 3(c), compared to the original image, added plant, removed a wall, and outpainted the image with ratio of 1.5, respectively, obtaining three images after visual dynamic strategies V1.
(2) Linguistically, described in section 4.3 we also simulate the language usage on LVLMs from different users. Specifically, when different users utilize LVLMs, they may have different linguistic expressions towards a same question due to their distinct identities or educational backgrounds. Therefore, as shown in Figure 3(a), our work conducts transformations on the original question from three linguistic levels: word-level (L1), sentence-level (L2), and context-level (L3, L4). For example in (gureB(c) the original question is 'What type of flooring does the kitchen have?'. From the word-level, some users might habitually use the word 'cookroom' instead of "kitchen'. From the sentence level, more literary users like writers, might phrase it 'is equipped with what kind of" instead of "have what type of". From the context level, more talkative users like teachers pretend to guide into the question by giving some scene description aforehead, which corresponds to our L3 strategy "adding relevant context'. Meanwhile, children users, who may lack mature logical thinking ability, might first ramble about unrelated gadgets in the image before posing his realistic question to LVLMs. This corresponds to 14 strategy 'adding irrelevant context'.
Generally, our dynamic strategies effectively simulate various real user interaction scenarios from both visual and linguistic perspectives, which is very significant for the practical application and widespread adoption of LVLMs.
1.12 MORE RESULTS ON DATA CONTAMINATION
Pretraining- Dataset
Benchmark
Vanilla
I.I
1.2
1,3
1.4
SEEDBench
0.6531
0.5197
0.4461
0.4286
0.4021
LAION (100M)
MMBench
0.8574
0.7846
0.7238
0.7165
0.6985
MMF
0.9572
0.8540
0.8338
0.7434
0.5528
SEEDBench
0.3540
0.2622
0,2060
0,2165
0,1912
CC3M (3M)
MMBench
0.6871
0.5327
0.4318
0,4018
0.4191
MME
0.6213
0.4984
0.5001
0.4314
0,3909
SEEDBench
0.9501
0.6952
0.6429
0.6354
0.5994
COCO-Caption (0.1M)
MMBench
0.8386
0.7410
0.6347
0.7143
0.7085
MME
0.8121
0.7227
0.6533
0.6892
0.7121
Table 14: The text contamination rate about the original, and L1, L2, L3, L4 strategy between benchmarks and pre- pre-training dataset.
Wolfe, Jeremy M. "Visual attention." Seeing (2000): 335-386.
Xu, Jiang, et al. "Language in context: emergent features of word, sentence, and narrative comprehension." *Neuroimage* 25.3 (2005): 1002-1015.
21

---

Published as a conference paper at ICLR 2025
We further detect the text contamination rates of the newly generated questions after strategy 11, 12,13.14, based on the MME, SEEDBench, and MMBench benchmarks. Together with the original text contamination rate among contaminated images for comparison, the experimental results are shown in the 1a As can be seen, the text contamination rates of our 11, 12,13,14 question variants have decreased three pretraining sets LAION(100M), CC3M(3M), and COCO-Caption (0.1M) These prove that our newly generated questions are not found in the pretraining data, successfully relieved the data contamination issue in existing static benchmarks.
A.13 COMPATIBILITY OF VLB ON OTHER LVLM TASKS
Since our method can dynamically transform both images and text to generate samples, believe our framework can also be applied to other tasks that VLMs can perform. Below, we conduct experiments common multi-modal task of LVLM: image caption. Firstly, select the famous image caption benchmark, COCO Captior] and randomly sample a 500 pairs subset. Secondly, we our three visual strategies to generate images COCO Caption: adding new objects(V1), ) removing existing ob))cts()22,, and outpainting(Vs). Next, for each newly generated image, we use GPT-4V generate corresponding captions, composing new <image, caption pairs. Finally, evaluate the original and the three dynamic COcO-caption variants via Evalkit on 6 popular VLMs.
Model
Vanilla
V22
TransCore-M
65.4
60.1
62.4
59.1
DeepSeek
58.6
46.1
51.0
45.2
IntemVil-2
63.8
59.7
65.9
60.3
XComposer2
70.1
65.4
68.6
67.6
Monkey-Chat
68.5
61.6
65.9
62.8
Qwen-VL-Chat
39.8
40.8
42.5
36.3
Table 15: The result of image caption task on the original, and V1, V2, V/3 strategy in COCO-Caption.
As can be seen from the table 15 our newly generated samples achieve similar results to the original dataset. And similar conclusions can be drawn, namely that strategy (V2) is simpler than (Vi) and (V/). This demonstrates that our dynamic framework can also effectively be applied to other multi- modal tasks, allowing a more flexible and comprehensive evaluation of LVLMs.
A.14 FAIL CASES BEFORE JUDGE MODULE
In our human verification process, observed that the VI strategy leads to a much higher fail- to-detect rate before passing the judge module. We thought this is primarily because, in small portion of generated images, the addition of objects causes partial occlusion of the core objects related to the question. We selected two fail cases from MMBench V strategy, with the types of being attribute comparison and celebrity recognition, respectively. As can be seen Figure [5] (a), the added laptop covers most of the cat, thus affecting the comparison of animal sizes. In Figure 15 (b), the added balloon slightly obscures the person's 's face, which might impact the recognition the person. Therefore, images with partial occlusions like those are filtered by the judge module, which reflects the rigor and effect of the judge module.
(rh: Attribute Comparison
(bi: Celebily Recogiitinn
Question: picture, bear dell and the other is Are
Question: the
E Met the some Cas 1 Jadge.
Namyy Receey banald
laptop
Correct
Add: balloon
Connect
Figure 15: (a) The added laptop covers most of the cat, thus affecting the comparison of animal sizes. (b) The added balloon slightly obscures the person's 's face, which might impact the recognition of the person.
Chen, Xinlei, et al. "Microsoft encn captions: Data collection and evaluation server." *arXiv preprint 15X (2015).
22


Reviews:
[{"review_id": "WgZpVhWLDT", "reviewer": "Reviewer_Rays", "summary": "The paper seeks to change the static nature and data contamination of benchmarks for vision-language models.  The paper introduces VLB: vision-langauge bootstrapping that dynamically generates new visual question answering samples via bootstrapping.  The goal is for the evaluation protocol to evolve with VLM capabilities.  The paper finds that existing VLMs struggle on the new benchmark.", "questions": "1. For example in figure 3(a), how is the visual attention and linguistic understanding converted into V1, V2, L1, L2 etc.?\n2. How is it verified that the generated questions are not found in the pretraining data? Does using the VLB method ensure that? This is not discussed in the analysis.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies an important research direction: existing benchmarks are static and because of large-scale pretraining data, it is hard to verify is some test data has leaked into the pretraining or training data. This makes evaluation difficult and the paper seeks to develop a new paradigm for evaluation.\n2. The idea of using insights from user interactions to inform the transformations V and L is interesting.  \n3. The experiments are comprehensive.", "weaknesses": "1. The role of user interaction is not defined in detail.  See Q1.\n2. Question rephrasing has been previously explored in several other works on VQA (eg. VQA Rephrasings dataset) or robustness work such as VQA-LOL, VQA-Subquestions and others. What is the overlap of the proposed work with those benchmarks?\n3. The work focuses only on VQA but there are several tasks that VLMs can perform.  Can the framework also handle capabilities that have to be evaluated without VQA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrZv4nRtkH", "reviewer": "Reviewer_RXK7", "summary": "This work proposes a new evaluation protocol, Vision-Language Bootstrapping (VLB), for comprehensively evaluating large vision-language models (LVLMs) with reduced data contamination and dynamic difficulty. Existing benchmarks like MM-Bench are constructed from fixed image-question pairs, which are partially observed in the training procedure of LVLMs. In VLB, both the image and the question can be modified to change the difficulty and avoid answering by memorization, while preserving the consistency with the original answer. Different operations in changing the image or question result in a series of difficulty levels. Extensive experiments are conducted to show performance change with VLB, through which the work poses new challenges for LVLMs.", "questions": "Please check the weakness section above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Thorough experiment results validate the dynamic evaluation protocol VLB. First, a judge model is introduced to ensure that the dynamic image-question pair is still consistent with the original answer. Second, human examination on 2,100 samples verifies that less than 5% samples would introduce inconsistency.\n\n2. The composition of multiple strategies effectively reduces data contamination and enables a wide range of difficulty levels. VLB can serve as a more reliable evaluation protocol than the traditional static ones.\n\n3. The new evaluation protocol can be readily combined with existing LVLM benchmarks.", "weaknesses": "1. The major concern lies in the performance variance. Even with the same image-question sample and the same bootstrapping strategy, different dynamic samples can be generated, due to the randomness in GPT-4V and PowerPaint. However, the experiments do not show the scale of this variance caused by randomness. If this variance is large, the performance metrics may be less reliable.\n\n2. Although the human verification (Figure 11) shows high consistency for each bootstrapping strategy, it is unclear if the consistency remains with composition of multiple strategies (e.g., V1+V3+L4). The errors may accumulate, and more changes to the image-question pair tend to break the original consistency.\n\n3. The bootstrapping strategies rely on GPT-4V and PowerPaint. If they are replaced/combined with models with similar functions, can similar observations remain?\n\n4. Some minor suggestions that do not affect the rating:\n    - The bootstrapping strategies may be reordered to reflect the difficulty level. For example, how about making \"removing existing objects\" V1 and \"adding new objects\" V3?\n    - In Figure 3(c), L3 should be \"add relevant context.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n8kuPaceeP", "reviewer": "Reviewer_7UPF", "summary": "This work introduces VLB, a benchmark generation strategy aiming to prevent the baseline models from being evaluated on contaminated data (data leakage in essence). VLB extends current LVLM benchmarks by employing bootstrapping strategies to create altered test cases in variable difficulties in a controllable manner. The experimental results using VLB-modified benchmarks show that data leakage is evidently prominent in existing practices, and VLB may help establish fairer baselines for LVLM evaluations.", "questions": "Here I also list down a few minor suggestions.\n\n1. The naming order of the VLB strategies could be changed so that the easy ones be introduced first. For example, it would feels more natural to first introduce L3, so that readers can more easily tell that, from Table 2, L3 is the positive alternative with additional helpful cues and lead to mostly improvement performance. The same idea applies for V2 as in Table 1.\n2. The term 'data contamination' is not a coined term. I sense the authors want to describe the phenomenon that pre-training data already entail the contents for the supposedly unknown test set cases. In fact, this already has had a name **data leakage** as in (Chen et al, 2024). Let's stick with the established terminology. But feel free to correct me if the authors believe the two have any nuanced difference.\n\nTypo: Table reference missing at Line 707.\n\nUpdate 11/23: Raising my assessment according to the authors' response.\n\nAfter all, I find this work very intriguing although the contribution feels a bit lackluster. However, I am open for reassessment after learning more from the exchange with the authors in the rebuttal period.\n\nChen et al, 2024. Are We on the Right Way for Evaluating Large Vision-Language Models?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is a pleasant read and is easy to follow. The paper is written in a well structured way following a clear plot line. I particularly find the parts where the bootstrapping strategies are introduced well written, which greatly helps me understand this work on an intuitive level.", "weaknesses": "I do have one particular concern regarding the veracity of the VLB-modified data.\n\nVLB strategies such as V1 (editing in a new object in the image) and L4 (adding irrelevant context in to the text) modify the original test case in a controlled manner. **However, how do we verify if the original test cases have been loyally modified in the way we want?** So far, such veracity verification steps are only observed in Figure 11 using human verification on a small batch of sampled data. The authors should consider additional automated verification techniques, such as the shift in semantic scores/image-text alignment metrics before vs after applying VLB strategies. \n\nAfter all, we do not want to re-evaluate LVLMs on wrongly generated test cases, which undermine the entire effort of VLB to start with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9VUpWclmsE", "reviewer": "Reviewer_jqNx", "summary": "The paper introduces a dynamic multimodal evaluation strategy to avoid data contamination for benchmarks. It shows that current popular evaluation benchmarks exist image-only contamination and image-text contamination. Image bootstrapping and language bootstrapping method can be used separately or in combination to adjust the difficulty of the questions while keeping the answers unchanged. Several experiments on 3 benchmarks and 8 VLMs are provided to verify the author's point of view.", "questions": "1. In practice, how can you ensure that when generating a lot of new samples, changes in images and questions do not introduce unnecessary bias or semantic errors?\n2. Are there any automated validation methods that can guarantee semantic consistency across all generated samples?\n3. When the external tools, such as PowerPaint, SAM and GPT-4V, being relied upon have limitations, for example, sometimes there will be some artifacts after using PowerPaint to remove an object, how does the DME ensure the fairness and accuracy of the evaluation?\n4. How to improve the interpretability of evaluation results?\n5. How to assist researchers in better understanding the specific areas where the model underperforms?\n6. Can you provide the specific VLMs mentioned in Table 1? For example, InternVL2-1B or InternVL2-2B or … InternVL2-76B.\n7. In Figure 4 “image outpainting” part, the proportion of the floor in the image V3 is larger than in the Vanilla image, and the question is “What type of flooring does the kitchen have?”, so in this situation, can “image outpainting” be defined as a hard task?\n8. Have you resized the image when using PowerPaint and GPT-4V?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The paper proposes a novel dynamic multimodal evaluation framework VLB, which has a a flexible complexity adjustment evaluation mechanism.\n2. By editing images and modifying questions, a set of evaluation samples with different levels of complexity can be generated. This method can be used to probe the upper and lower bounds of VLMs' capabilities in certain tasks.\n3. The VLB framework is highly versatile. It can be easily plugged in a lot of benchmarks.", "weaknesses": "1. Authors use many models, such as GPT-4V, PowerPaint, SAM, to insert and remove objects from original images and employ GPT-4 for language reconstruction. Each step requires consistency evaluation, which may increase the complexity of implementation and lead to more unexpected errors or inconsistencies.\n2. Generating new visual and language samples takes a long time, especially during large-scale evaluations. It also requires significant computational resources to produce new samples.\n3. The VLB framework relies on PowerPaint, SAM and GPT-4V to generate new visual and language samples.\n4. Although DME can generate diverse samples, the performance of VLMs may decrease on these samples, and the specific reasons for this are not clear.\n5. Lack visualization for error analyses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)