Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping

## Abstract

Abstract Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across multimodal tasks such as visual perception and reasoning, leading to good performance on various multimodal evaluation benchmarks. However, these benchmarks keep a static nature and overlap with the pre-training data, resulting in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce a dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MMBench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.

## 1 Introduction

Large Vison-Language Models (LVLMs) (Achiam et al., ; Lu et al., ; Research, ; Chen et al., ; Liu et al., ) have achieved unprecedented performance
across a wide range of multimodal tasks such as visual perception (Goyal et al., ) and commonsense reasoning (Zellers et al., ).
The impressive progress necessitates the creation of nuanced evaluations to track LVLM development and understand the capability boundary of LVLMs. It leads to the advancement of a series of evaluation benchmarks such as SEEDBench (Li et al., ), MMBench (Liu et al., ), and MME (Fu et al., ). The evaluation results are crucial in selecting suitable LVLM for various applications.

Despite the proliferation of LVLM evaluations, there are increasing concerns about the genuine capabilities of LVLMs (Laskar et al., ), largely due to two key challenges associated with current evaluation benchmarks.
1) Data contamination.
LVLMs are pre-trained on large datasets, often sourced from the internet. Unfortunately, many evaluation benchmarks are constructed from similar sources, leading to a high likelihood of overlap with training data, thus causing data contamination (Touvron et al., ; Chen et al., ), as illustrated in Figure 1(a) and detailed in Section 3.
It raises a critical concern: “Does the model genuinely perceive and understand the input, or is it merely memorizing it?”
2) Static dataset with fixed complexity.
As shown in Figure 1(b), existing benchmarks for LVLMs are manually collected (Xu et al., ; Li et al., ). Once constructed, they are static with a fixed complexity, making them inadequate to keep pace with the rapid development of LVLMs.
To accurately assess LVLM performance boundaries, a dynamic, automated evaluation protocol with adjustable complexity is urgently needed.

Figure: Figure 1: (a) shows that some images in evaluation sets can be exactly found in the training set and their corresponding questions can be solved by the captions of similar training images. (b) compares our dynamic multimodal evaluation with the previous static evaluation. We can see that dynamic evaluation can create various variants upon static benchmarks with flexible complexity.
Refer to caption: x1.png

However, it is challenging to develop a dynamic evaluation framework for LVLMs. Existing dynamic evaluation protocols apply to only Large Language Models (LLMs). For example,
DyVal (Zhu et al., ) dynamically synthesizes test samples for LLMs based on directed acyclic graphs to combat data contamination. NPHardEval (Fan et al., ) generates new test samples for NP-hard math problems. Yet, the reliance on graph structure and math knowledge limits the applicability of these methods.
Recently, MPA (Zhu et al., ) dynamically creates new questions from the old ones for evaluating LLMs. However, building visual dynamics based on existing benchmarks requires manipulating the visual contents in the image while maintaining its essence, which poses a great challenge and remains unexplored.

In this work, we develop a dynamic multimodal evaluation (DME) protocol with flexible complexity by proposing Vision-Language Bootstrapping (VLB). VLB consists of a multimodal bootstrapping module and a judge module. The multimodal bootstrapping dynamically creates new visual question-answering (VQA) samples through various image and language bootstrapping strategies. These carefully designed strategies do not rely on predefined rules, making them applicable to various multimodal tasks. Meanwhile, the judge ensures that the newly generated samples maintain consistency with the original ones, *e.g.* preserving the correctness of the answer after bootstrapping.

To dynamically create new evaluation suites with flexible complexity, we propose various bootstrapping strategies with complexity control for both image and question. These strategies stem from practical situations in real user interactions, primarily simulating user diversity in two aspects: different visual attention and linguistic understanding. By bootstrapping a VQA sample with various transformations flexibly, we can build a series of dynamic variants with diverse complexities upon an existing benchmark. Experimental results show that the data contamination issues can also be remarkably reduced on these dynamic variants. Therefore, the proposed VLB can provide a comprehensive and robust assessment of LVLM capabilities, ensuring that the evaluation co-evolves with the ever-evolving abilities of LVLMs.

Our VLB is general enough to be applied to various existing benchmarks. In experiments, we employ VLB to bootstrap several representative evaluation benchmarks such as SEEDBench (Li et al., ) and MMbench (Liu et al., ). We evaluate various LVLMs including closed-source APIs (GPT-4o (OpenAI, ), Claude3-Sonet (Anthropic, )), and $9$ popular open-source LVLMs such as DeepSeek-VL (Lu et al., ) and Yi-VL (Young et al., ). The takeaways of our key findings are as follows:

- •
All LVLMs exhibited performance decreases on our multimodal dynamic benchmarks, indicating existing limitations in current LVLMs towards adapting to different users with various identities. (Section 5.2 and 5.3)
- •
Through image bootstrapping strategies, we observed that distracting or focusing visual attention on the image will affect LVLMs’ capabilities to understand the key image content related to the corresponding question. (Section 5.2)
- •
By employing language bootstrapping strategies on existing benchmarks, we found that current LVLMs exhibit increasing sensitivity as modifications escalate from words to sentences and finally to context changes. (Section 5.2)
- •
VLB can provide multiple dynamic benchmarks with flexible complexity via multimodal and multi-strategy composition. By combining more difficult strategies, we can increase the complexity of existing benchmarks, posing significant challenges for LVLMs, including advanced models like InternVL-2 and GPT-4o. (Section 5.3)
- •
LVLMs exhibit varying performance changes on our dynamic strategies across various tasks, especially sensitive to ‘Instance Interaction’, ‘Text Understanding’, and ‘Spatial Relation’ tasks. (Section 5.4)

Overall, the contributions of this paper are summarized as follows. 1) We delve into the data contamination issue of existing multimodal evaluation benchmarks and find a pronounced overlap between evaluation samples and pre-training data, which hinders the assessment of previous static benchmarks. 2) We propose a dynamic multimodal evaluation protocol called vision-language bootstrapping (VLB). VLB can evolve existing benchmarks with visual and linguistic dynamics, obtaining various variants with flexible complexity and reduced data contamination. 3) We perform comprehensive evaluations on a variety of popular LVLMs, indicating that the existing LVLMs still struggle to adapt to different user interactions and intent. Our code will be available at [https://github.com/yangyue5114/DME](https://github.com/yangyue5114/DME).

## 2 Related Work

Data contamination.
Data contamination raises widespread concerns across evaluation benchmark datasets in both LLMs and LVLMs. Study (Dodge et al., ) reveals exact match contamination rates ranging from 2% to 50% across various benchmarks when compared to C4 pretraining data. Llama-2 (Touvron et al., ) reported over 16% of MMLU examples are contaminated, where 11% are seriously contaminated. Subsequently, the concern of data contamination extends to LVLMs. Recent work (Chen et al., ) find Sphinx-X-MoE (Gao et al., ) gets 43.6% on MMMU (Yue et al., ) without accessing images, surpassing its LLM backbone with 17.9%, meaning potential contamination of multimodal benchmarks. In section 3, we first verify a high rate of contamination in both visual and linguistic modalities, between static multimodal benchmarks and training sets. This severe contamination necessitates the development of dynamic evaluation to advance LVLMs more accurately.

LVLM Evaluation.
With the rapid development of LVLMs, various benchmarks are proposed for evaluating LVLMs. Early single-task benchmarks, like ok-VQA (Marino et al., ), and MS-COCO (Chen et al., ), are proposed to evaluate coarse-grained ability. Later, current LVLM evaluation benchmarks aimed to provide relatively holistic evaluations for the overall reasoning capabilities of LVLMs, such as LVLM-eHub (Xu et al., ), MME (Yin et al., ), SEEDBench (Li et al., ), MM-Vet (Yu et al., ), MMBench (Liu et al., ), and Llavebench (Liu et al., ). However, the static nature of current benchmarks makes them susceptible to contamination and restricts them to a fixed level of complexity. Thus, there is an urgent need for dynamic evaluation methods to enable comprehensive assessments of LVLMs.

Dynamic Evaluation.
Recently, some researchers have pioneered the exploration of dynamic evaluation. Study (Zhu et al., ) proposed DyVal to dynamically generate test samples based on the graph structure to combat data contamination. Similarly, NPHardEval (Fan et al., ) generates new evaluation samples for NP-hard mathematical problems. To extend dynamic evaluation to more diverse NLP tasks, Zhu et al. () further developed MPA, which employs LLM-based agents to transform existing problems into new ones. However, these dynamic evaluations only focus on LLMs and NLP benchmarks, leaving the dynamic evaluation of LVLMs unexplored. Our work is the first to design strategies for bootstrapping both visual images and language questions, to achieve dynamic evaluation on LVLMs. Based on an original benchmark, we can generate multiple variants with flexible complexity and lower contamination rates.

Figure: Figure 2: (a) Existing benchmarks have severe overlap on images with pre-training data. (b) Questions of the contaminated evaluation image can also be solved by the caption of similar images from the training set.
Refer to caption: x2.png

## 3 Revisiting Data Contamination in Existing Benchmarks

We explore two types of data contamination in multimodal evaluation benchmarks. 1) Image-only contamination. We aim to detect how many images in the benchmark can be found in the pre-training data. To this end, we utilize the CLIPScore (Hessel et al., ) to measure the similarity between images from the evaluation and training set. In our pilot experiments, we find that if the CLIPScore between two images exceeds 0.9, it indicates high visual similarity. Thus, we adopt 0.9 as the threshold to determine visual contamination. The image-only contamination rate is calculated as the ratio of the number of contaminated images and the number of total images in the evaluation set.
2) Image-text contamination. Beyond images, the question and answer of benchmark can also be contaminated. We extend ideas from NLP detection works (Li et al., ) to identify this image-text contamination. For contaminated image pairs, we determine the question and answer contaminated if the answer can be directly inferred from the captions of the training image. In practice, we leverage GPT-4 to conduct this process.

As illustrated in Figure 2, we examine the two types of data contamination between three popular evaluation benchmarks: SEEDBench, MMBench, MME, and three widely used pre-training sets: LAION-100M (Schuhmann et al., ), CC3M (Changpinyo et al., ), COCO-Caption (Chen et al., ). The results reveal that each evaluation benchmark presents certain contamination rates across training datasets of various sizes, even with some reaching as high as 84.46% (image-only) and 33.13% (image-text). Note that the actual size of pre-training data far exceeds our detected maximum of 100M, which indicates that the actual contamination issue could be even more severe.

## 4 Dynamic Multimodal Evaluation

This section introduces our dynamic multimodal evaluation framework, Vision-Language Bootstrapping (VLB). Section 4.1 gives an overview. Section 4.2 and 4.3 present one component of VLB, multimodal dynamics: mage bootstrapping and language bootstrapping. The other component, the judge module, is in Section 4.4. The compositional bootstrapping is described in Section 4.5.

Figure: Figure 3: Illustration of our proposed dynamic multimodal evaluation framework, Vision-Language Bootstrapping (VLB). (a) demonstrates how we derive insights from real user interactions with LVLMs, where users possess different visual attention and language understanding from diverse identities. (b) highlights the role of VLB’s judge module in ensuring that generated images and questions maintain consistent with the original. (c) provides an example of VLB transforming a sample through image and language bootstrapping. Additionally, VLB can generate new, increasingly complex samples through bootstrapping composition.
Refer to caption: x3.png

### 4.1 Overview

Formulation of Dynamic Multimodal Evaluation (DME). Let us begin with defining a dynamic multimodal evaluation for LVLMs. Given an original VQA sample $E_{s}=(I,Q,A)$ where $I,Q$ and $A$ denote the image, question, and ground truth answer, a dynamic sample can be represented as $E_{d}=(\mathcal{V}(I),\mathcal{L}(Q),A)$. Here we use the subscripts ‘$s$’ and ‘$d$’ to denote the original static and dynamic evaluation samples, respectively. $\mathcal{V}$ and $\mathcal{L}$ are transformations applied to the image and question. By comparing $E_{s}$ with $E_{d}$, we see that DME is implemented by image and language bootstrapping while preserving the consistency with the original sample.

Challenges. There are three major challenges in achieving dynamic multimodal evaluation. First, the DME framework should be generic enough to be used in various multimodal tasks instead of limited to a specific task like (Fan et al., ). Second, it is hard to alter images and questions while not affecting the correctness of the reference answer. In linguistics, GPT4 can be used as a powerful agent to rewrite questions while preserving the consistency with the original question by following instructions (Zhu et al., ). However, there is no agent to achieve this in image transformation. Third, the DME framework should be flexible enough to control the complexity of newly generated evaluation samples, enabling the evaluation to co-evolve with LVLM development.

Our Approach. We propose a novel DME protocol, called vision-language bootstrapping (VLB). VLB consists of a multimodal bootstrapping module to generate new images and questions, and a judge module to maintain consistency with the original sample. As illustrated in Figure 3 (a), by simulating real LVLM’s user interaction in visual attention and linguistic understanding, we design image (*i.e.*, $\mathcal{V}\in\{\mathcal{V}_{1},\mathcal{V}_{2},\cdots\}$) and language (*i.e.*, $\mathcal{L}\in\{\mathcal{L}_{1},\mathcal{L}_{2},\cdots\}$) bootstrapping strategies. Experiments show that the composition of $\mathcal{V}$ and $\mathcal{L}$ would yield dynamic evaluation samples with flexible complexity, an example is exhibited in Figure 3 (c).

### 4.2 Image Bootstrapping

Our VLB employs image bootstrapping to build visual dynamics. Since different users pose various identities and backgrounds, their levels of visual attention also vary. For instance, children’s attention might be more easily distracted by irrelevant objects in images than educated adults. Thus, we simulate disturbing or focusing visual attention for image bootstrapping: adding new objects, removing existing objects, and expanding original images, as Figure 4. Each strategy is associated with a difficulty level (*i.e.* ‘Easy’ or ‘Hard’).

$\mathcal{V}_{1}$: Adding New Objects (Hard). Given the original sample $E_{s}=(I,Q,A)$, $\mathcal{V}_{1}$ aims to insert a new object into the image $I$ while maintaining that the answer $A$ is still correct for generated image $\mathcal{V}_{1}(I)$. To this end, we integrate GPT-4V (Achiam et al., ) and the advanced image editing model PowerPaint (Zhuang et al., ) in two steps. First, GPT-4V is prompted with a carefully designed add-instruction (detailed in Appendix A.1) to return a set of insertable objects. Each object is in the form of (object name: bounding box). Second, we transformed the bounding box into mask, which together with the object name and original image $I$, are fed into the PowerPaint to generate the new image $\mathcal{V}_{1}(I)$. Adding objects into an image introduces irrelevant visual content, which would distract LVLMs and may be harder than the vanilla sample $E_{s}$.

$\mathcal{V}_{2}$: Removing Existing Objects (Easy). Simplifying the visual content by removing objects poses a substantial challenge because GPT-4V is relatively weak in fine-grained visual grounding. To tackle this problem, we extend the idea of SoM (Yang et al., ) that unleashes the visual grounding capabilities of GPT-4V by providing semantic marks. Leveraging SAM (Kirillov et al., ), we obtain semantic masks $\mathbf{M}$ of all objects within an image. We then assign each object with a serial number. Afterward, we prompt GPT-4V with remove-instruction to give the removable objects, each formatted as (object name, serial number). Finally, the image $I$, object name, and mask corresponding to the serial number are fed into PowerPaint (Zhuang et al., ) to obtain the new image. Since removing objects from the image would omit superfluous visual content, $E_{d}$ would be easier than the vanilla $E_{s}$ under $\mathcal{V}_{2}$.

$\mathcal{V}_{3}$: Expanding Original Images (Hard). $\mathcal{V}_{3}$ employs image outpainting to extend the background of the image while not affecting the core elements within the image relevant to the question $Q$. Since expanding the image also introduces irrelevant visual content to the problem, $E_{d}$ would be more challenging than the vanilla $E_{s}$ under $\mathcal{V}_{3}$. We set the extension ratio $r=1.5$ for the main experiments. Intuitively, varying the outpainting ratio $r$ mimics the effect of observing objects from different distances. We provide an ablation study on the effect of outpainting ratios in Section 5.4.

Figure: Figure 4: Strategies of image bootstrapping: Starting from an original image, route $\mathcal{V}_{1}$, $\mathcal{V}_{2}$, $\mathcal{V}_{3}$ represents the process of adding new objects, removing existing objects, and expanding original images, respectively.
Refer to caption: x4.png

### 4.3 Language Bootstrapping

Besides visual modality, language understanding is also crucial for LVLMs. We construct language bootstrapping by stimulating different linguistic expressions of users with various identities and backgrounds. Four strategies from different levels of linguistic expressions: word level, sentence level, context level, are employed.
Figure 3 illustrates the transformed samples of four paradigms ${\mathcal{L}_{1},\mathcal{L}_{2},\mathcal{L}_{3},\mathcal{L}_{4}}$ based on the original question $Q$. We also assigned a difficulty level with ‘Easy’ or ‘Hard’ for each language bootstrapping strategy.

$\mathcal{L}_{1}$: Word Substitution (Hard). Given that each user of LVLMs possesses unique vocabulary preferences and habits, $\mathcal{L}_{1}$ simulate these variations by replacing words with synonyms or contextually similar expressions while preserving the core concept for question $Q$ by following (Zhu et al., ). LLM pre-training usually has limited coverage of phrasing styles in instructions. Therefore, $\mathcal{L}_{1}$ would increase the difficulty of the original problem.

$\mathcal{L}_{2}$: Sentence Rephrasing (Hard). Considering users with different identities might express the same question with various phrasing styles. For example, the question “How does photosynthesis work?” would become “How do plants make food from sunlight?” for a casual user, while turning into “What are the key differences between the light-dependent and light-independent reactions in photosynthesis?” for a researcher. To this end, $\mathcal{L}_{2}$ employs role-playing (Wang et al., ) to rephrase the original question $Q$ into new ones.

$\mathcal{L}_{3}$: Adding Relevant Context (Easy).
$\mathcal{L}_{3}$ aims to introduce visual-related text to the original question $Q$. The generated context should contain more hints to help understand the image. However, it should be ensured that the original question cannot be solved directly with the added context. The strategy reflects the situation when users understand the image partly. We employ GPT-4V to generate descriptions of the image via our carefully designed prompts (Appendix A.1). Then we add the generated description to $Q$, obtaining the new question $\mathcal{L}_{3}(Q)$. By design, the dynamic variant $E_{d}$ would be easier than the original $E_{s}$.

$\mathcal{L}_{4}$: Adding Irrelevant Context (Hard).
Unlike $\mathcal{L}_{3}$, $\mathcal{L}_{4}$ will supplement the question $Q$ with visual-related text which cannot help answer the question. Here we also use GPT-4V to create context. Since the irrelevant context would distract LVLMs, the new sample $E_{d}$ would become harder than the original $E_{s}$.

### 4.4 Judge Module

DME should create new samples while maintaining essential concepts of the original one.
Although our VLB employs strict criteria in the bootstrapping process, the transformation may still break the consistency. For instance, the added objects in $\mathcal{L}_{1}$ might shelter key visual content relevant to the problem.
To tackle this issue, we incorporate a judge module to ensure consistency between the dynamic variant and the original sample.

Specifically in Figure 3(b), given the static sample $E_{s}=(I,Q,A)$ and the corresponding dynamic variant $E_{d}=(\mathcal{V}(I),\mathcal{L}(Q),A)$, the judge module is informed what modification is conducted by $\mathcal{V}(I)$ and $\mathcal{L}(Q)$, and is asked to check whether $A$ is still correct for the newly generated image and question. Here we designed different judge prompts to check each bootstrapping strategy, detailed prompts are in Appendix A.2.
Similar to MPA (Zhu et al., ), the judge operates in an adversarial manner and returns a ‘Yes’ or ‘No’ verdict. If the response is ‘No’, the sample will be regenerated until it passes the assessment. If the sample does not pass after five attempts, the original sample is used instead.
In practice, we use InternVL-2, an open-source powerful LVLM (Chen et al., ; ), as a capable and affordable judge. What’s more, we further validate the effectiveness of our judge module by the human verification in Section A.5

### 4.5 Compositional Bootstrapping

Due to every single VLB strategy for image and question being atomic, we can investigate two kinds of bootstrapping composition with flexible complexities. 1) Paired multimodal composition. We can compose visual bootstrapping $\mathcal{V}$ and linguistic bootstrapping $\mathcal{L}$, into a paired multimodal dynamic sample $E_{d}=(\mathcal{V}_{i}(I),\mathcal{L}_{j}(Q),A)$, obtaining a total of $12$ dynamic variants. 2) Multi-strategy composition. We can also stack multiple image bootstrapping strategies on a single image or multiple language bootstrapping strategies on the question, composing a multi-strategy dynamic sample like $E_{d}=(\mathcal{V}_{3}(\mathcal{V}_{1}(I)),\mathcal{L}_{4}(Q),A)$.

Since each single VLB strategy possesses different levels of complexity , the above two kinds of compositions can effectively construct different variants varying in complexity, to assess the robustness and adaptability of LVLMs and explore models’ upper and lower limits in performance across different benchmarks.

## 5 Experiment

### 5.1 Experiment Setup

Tasks and Datasets. We selected five popular benchmark datasets to assess current LVLMs, encompassing Yes/No Questions (MME), Multiple Choice Questions (MMBench, SEEDBench), and Visual Question Answering (MMvet, LLaVABench). These benchmarks include a broad spectrum of cognitive and comprehension tasks. In Section 5.2 and 5.3, we employ three comparable datasets in terms of size: MME, MMBench (30%), and SEEDBench (10%) as the experimental datasets. Then we extend our dynamic strategies to the full set of MMBench, MMvet, and LLaVABench in Section 5.3

Evaluated LVLMs. Our evaluated LVLMs include both closed-source APIs: GPT-4o (OpenAI, ) and Claude (Anthropic, ), and open-source models: DeepSeek-VL (Lu et al., ), TransCore_M (Research, ), XComposer2 (Dong et al., ), Monkey-Chat (Li et al., ), LLaVA-NeXT-Vicuna (Liu et al., ), Yi-VL-34B (Young et al., ), Qwen-VL (Bai et al., ) and InternVL (Chen et al., ). We utilize the standardized evaluation platform VLMEvalkit (Duan et al., ) and set the generation temperature as 0 for all evaluated LVLMs to ensure a fair comparison.

### 5.2 result of single bootstrapping strategy

The results of image bootstrapping strategies.
We apply our designed image bootstrapping strategies: Adding New Objects ($\mathcal{V}1$), Remove Existing Objects ($\mathcal{V}2$), and Expanding Original Images ($\mathcal{V}3$) on existing benchmarks. Table 1 presents the evaluation results across a series of LVLMs.
Consistent with predefined difficulty levels, $\mathcal{V}_{1}$ and $\mathcal{V}_{3}$ generally result in a decrease in accuracy, and $\mathcal{V}2$ slightly boost the performance.
It indicates that adding irrelevant objects or expanding the original image can introduce more superfluous visual content, thereby hindering LVLMs from capturing the key visual content related to the problem. Meanwhile, removing some irrelevant, extraneous objects can simplify the visual information of the image, allowing LVLMs to better focus on the key elements. More visualized cases are in Appendix A.3 and A.4.

**Table 1: Results of three image bootstrapping strategies on LVLMs. For each benchmark, ‘Vanilla’ represents the original results, while $\mathcal{V}_{1}$, $\mathcal{V}_{2}$, and $\mathcal{V}_{3}$ represent the results after image bootstrapping. (The numbers in parentheses show the difference compared to vanilla, with red indicating decreasing, and green increasing.)**
| Model | SEEDBench (%) | MMBench (%) | MME (%) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vanilla | $\mathcal{V}$1 | $\mathcal{V}$2 | $\mathcal{V}$3 | Vanilla | $\mathcal{V}$1 | $\mathcal{V}$2 | $\mathcal{V}$3 | Vanilla | $\mathcal{V}$1 | $\mathcal{V}$2 | $\mathcal{V}$3 |  |
| DeepSeek-VL | 69.44 | 64.79 (4.65$\downarrow$) | 70.35 (0.91$\uparrow$) | 69.39 (0.05$\downarrow$) | 79.72 | 75.90 (3.82$\downarrow$) | 80.03 (0.31$\uparrow$) | 77.96 (1.76$\downarrow$) | 86.08 | 78.16 (7.92$\downarrow$) | 86.59 (0.51$\uparrow$) | 82.69 (3.39$\downarrow$) |
| TransCore-M | 73.58 | 69.10 (4.48$\downarrow$) | 73.86 (0.28$\uparrow$) | 69.52 (4.06$\downarrow$) | 79.64 | 75.37 (4.27$\downarrow$) | 79.72 (0.08$\uparrow$) | 76.89 (2.75$\downarrow$) | 88.19 | 83.13 (5.06$\downarrow$) | 88.87 (0.68$\uparrow$) | 86.15 (2.03$\downarrow$) |
| Monkey-Chat | 69.58 | 65.04 (4.54$\downarrow$) | 70.92 (1.34$\uparrow$) | 68.04 (1.54$\downarrow$) | 80.26 | 75.37 (4.89$\downarrow$) | 80.29 (0.03$\uparrow$) | 79.11 (1.15$\downarrow$) | 86.34 | 79.59 (6.75$\downarrow$) | 87.09 (0.75$\uparrow$) | 84.03 (2.31$\downarrow$) |
| LLaVA-NeXT-Vicuna | 71.54 | 65.60 (5.94$\downarrow$) | 71.88 (0.34$\uparrow$) | 67.83 (3.71$\downarrow$) | 79.26 | 74.04 (5.22$\downarrow$) | 79.27 (0.01$\uparrow$) | 77.74 (1.52$\downarrow$) | 68.97 | 59.61 (9.36$\downarrow$) | 69.39 (0.42$\uparrow$) | 68.46 (0.51$\downarrow$) |
| Qwen-VL-Chat | 69.58 | 65.04 (4.54$\downarrow$) | 70.92 (1.34$\uparrow$) | 68.04 (1.54$\downarrow$) | 71.84 | 67.31 (4.53$\downarrow$) | 72.37 (0.53$\uparrow$) | 72.69 (0.85$\downarrow$) | 74.36 | 67.03 (7.32$\downarrow$) | 78.16 (3.79$\uparrow$) | 81.73 (7.37$\uparrow$) |
| XComposer2 | 75.40 | 69.18 (6.22$\downarrow$) | 75.19 (0.21$\downarrow$) | 71.13 (4.27$\downarrow$) | 84.62 | 77.23 (7.39$\downarrow$) | 85.19 (0.57$\uparrow$) | 76.89 (2.75$\downarrow$) | 92.32 | 89.20 (3.11$\downarrow$) | 93.25 (0.93$\uparrow$) | 94.03 (1.71$\uparrow$) |
| Yi-VL-34B | 68.25 | 62.84 (5.41$\downarrow$) | 67.69 (0.56$\downarrow$) | 65.24 (3.01$\downarrow$) | 81.63 | 74.49 (6.83$\downarrow$) | 81.46 (0.17$\downarrow$) | 79.57 (2.06$\downarrow$) | 80.86 | 73.25 (7.60$\downarrow$) | 82.63 (1.76$\uparrow$) | 76.53 (4.32$\downarrow$) |
| InternVL-2 | 76.80 | 70.46 (6.34$\downarrow$) | 77.67 (0.87$\uparrow$) | 74.45 (2.34$\downarrow$) | 88.59 | 80.33 (8.26$\downarrow$) | 89.28 (0.68$\uparrow$) | 83.01 (5.57$\downarrow$) | 80.01 | 72.70 (7.31$\downarrow$) | 82.31 (2.29$\uparrow$) | 76.53(3.48$\downarrow$) |
| GPT-4o | 77.59 | 70.91 (6.68$\downarrow$) | 78.11 (0.52$\uparrow$) | 73.70 (3.89$\downarrow$) | 87.37 | 82.28 (6.68$\downarrow$) | 89.06 (1.68$\uparrow$) | 85.28 (2.09$\downarrow$) | 77.57 | 71.04 (6.52$\downarrow$) | 78.49 (0.92$\uparrow$) | 72.30 (5.26$\downarrow$) |

**Table 2: Results of the vanilla and employing language bootstrapping strategy $\mathcal{L}_{1}$, $\mathcal{L}_{2}$, $\mathcal{L}_{3}$, $\mathcal{L}_{4}$ on LVLMs.**
| Model | SEEDBench (%) | MMBench (%) | MME (%) |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vanilla | $\mathcal{L}$1 | $\mathcal{L}$2 | $\mathcal{L}$3 | $\mathcal{L}$4 | Vanilla | $\mathcal{L}$1 | $\mathcal{L}$2 | $\mathcal{L}$3 | $\mathcal{L}$4 | Vanilla | $\mathcal{L}$1 | $\mathcal{L}$2 | $\mathcal{L}$3 | $\mathcal{L}$4 |  |
| DeepSeek-VL | 69.44 | 68.67 (0.77$\downarrow$) | 68.51 (0.93$\downarrow$) | 71.17 (1.73$\uparrow$) | 70.16 (0.72$\uparrow$) | 79.72 | 78.15 (1.57$\downarrow$) | 78.92 (0.80$\downarrow$) | 84.26 (4.54$\uparrow$) | 79.07 (0.65$\downarrow$) | 86.08 | 85.67 (0.41$\downarrow$) | 84.05 (2.03$\downarrow$) | 92.23 (6.25$\uparrow$) | 71.18 (14.90$\downarrow$) |
| TransCore_M | 73.58 | 72.94 (0.64$\downarrow$) | 71.67 (1.91$\downarrow$) | 71.95 (1.63$\downarrow$) | 72.19 (1.39$\downarrow$) | 79.64 | 78.53 (1.11$\downarrow$) | 78.62 (1.02$\downarrow$) | 84.89 (5.25$\uparrow$) | 78.55 (1.09$\downarrow$) | 88.19 | 85.57 (2.62$\downarrow$) | 86.31 (1.88$\downarrow$) | 87.42 (0.77$\downarrow$) | 72.92 (15.27$\downarrow$) |
| Monkey-Chat | 69.58 | 67.41 (2.17$\downarrow$) | 69.16 (0.42$\downarrow$) | 70.60 (1.02$\uparrow$) | 67.95 (1.63$\downarrow$) | 80.26 | 78.59 (1.67$\downarrow$) | 79.41 (0.85$\downarrow$) | 83.63 (3.37$\uparrow$) | 78.00 (2.26$\downarrow$) | 86.34 | 89.86 (3.52$\uparrow$) | 83.97 (2.37$\downarrow$) | 95.15 (8.81$\uparrow$) | 79.43 (6.91$\downarrow$) |
| LLaVA-NeXT-Vicuna | 71.54 | 70.16 (1.38$\downarrow$) | 69.86 (1.68$\downarrow$) | 73.06 (1.52$\uparrow$) | 70.61 (0.93$\downarrow$) | 79.26 | 78.84 (0.42$\downarrow$) | 77.90 (1.36$\downarrow$) | 83.80 (4.54$\uparrow$) | 77.14 (2.12$\downarrow$) | 68.97 | 65.73 (3.24$\downarrow$) | 69.97 (1.00$\uparrow$) | 80.49 (11.52$\uparrow$) | 63.43 (5.54$\downarrow$) |
| Qwen-VL-Chat | 63.62 | 60.01 (3.61$\downarrow$) | 60.84 (2.78$\downarrow$) | 61.05 (2.57$\downarrow$) | 52.98 (10.64$\downarrow$) | 71.84 | 68.29 (3.55$\downarrow$) | 69.73 (2.11$\downarrow$) | 69.54 (2.30$\downarrow$) | 61.32 (10.52$\downarrow$) | 74.36 | 39.28 (35.08$\downarrow$) | 63.51 (10.85$\downarrow$) | 79.21 (4.85$\uparrow$) | 54.18 (20.18$\downarrow$) |
| XComposer2 | 75.40 | 73.89 (1.51$\downarrow$) | 73.38 (2.02$\downarrow$) | 75.90 (0.50$\uparrow$) | 74.34 (1.06$\downarrow$) | 84.62 | 83.80(0.82$\downarrow$) | 84.04 (0.58$\downarrow$) | 88.57 (3.95$\uparrow$) | 83.64 (0.98$\downarrow$) | 92.32 | 88.35 (3.97$\downarrow$) | 91.12 (1.20$\downarrow$) | 92.45 (0.13$\uparrow$) | 73.74 (18.58$\downarrow$) |
| Yi-VL-34B | 68.25 | 66.34 (1.91$\downarrow$) | 66.30 (1.95$\downarrow$) | 71.49 (3.24$\uparrow$) | 66.70 (1.55$\downarrow$) | 81.63 | 80.39 (1.24$\downarrow$) | 80.66 (0.97$\downarrow$) | 86.39 (4.76$\uparrow$) | 78.96 (2.67$\downarrow$) | 80.86 | 82.74 (1.88$\uparrow$) | 79.66 (1.20$\downarrow$) | 89.21 (8.35$\uparrow$) | 65.09 (15.77$\downarrow$) |
| InternVL-2 | 76.80 | 75.26 (1.53$\downarrow$) | 74.77 (2.03$\downarrow$) | 77.24 (0.43$\uparrow$) | 74.31 (2.48$\downarrow$) | 88.59 | 85.38 (3.21$\downarrow$) | 87.52 (1.07$\downarrow$) | 89.59 (1.00$\uparrow$) | 85.53 (3.06$\downarrow$) | 80.01 | 77.15 (2.85$\downarrow$) | 79.34 (0.67$\downarrow$) | 79.52 (0.49$\downarrow$) | 69.98 (10.03$\downarrow$) |
| GPT-4o | 77.59 | 77.01 (0.57$\downarrow$) | 75.89 (1.70$\downarrow$) | 78.29 (0.70$\uparrow$) | 74.69 (2.90$\downarrow$) | 87.37 | 86.61 (0.76$\downarrow$) | 85.84 (1.53$\downarrow$) | 87.60 (0.99$\uparrow$) | 81.10 (5.51$\downarrow$) | 77.57 | 76.22 (1.34$\downarrow$) | 70.99 (6.57$\downarrow$) | 78.03 (0.46$\uparrow$) | 73.35(4.22$\downarrow$) |

The results of language bootstrapping strategies.
We also apply linguistic bootstrapping on different linguistic levels, including word substitution ($\mathcal{L}1$), sentence rephrasing ($\mathcal{L}2$), relevant context($\mathcal{L}3$), and irrelevant context ($\mathcal{L}4$). Table 2 illustrates that, except for $\mathcal{L}_{3}$, the other strategies result in a degradation of LVLM performance. This highlights the challenge LVLMs face in addressing questions posed by different individuals from diverse backgrounds. Therefore, further efforts are needed to enhance LVLMs’ ability to comprehend the semantic essence of questions and improve their user adaptability. These findings align with our previous revisiting experiments, suggesting that benchmarks may be contaminated. Conversely, incorporating relevant captions $\mathcal{L}3$ helps the model better understand images, boosting accuracy. Quantitatively, employing $\mathcal{L}3$ and $\mathcal{L}4$ presents more fluctuation than $\mathcal{L}1$ and $\mathcal{L}2$. This may be due to the situation (Kamradt, ) that longer text poses a greater influence for LVLMs compared to shorter words and sentence.

Figure: Figure 5: Results of composing image and language bootstrapping strategies on MMBench.
Refer to caption: x5.png

**Table 3: Results of the hardest and easiest variants after multimodal bootstrapping strategy composition. For each benchmark, ‘vanilla’ represents the original results, while the hardest is $\mathcal{V}_{1}$ + $\mathcal{L}_{4}$, easiest is $\mathcal{V}_{2}$ + $\mathcal{L}_{3}$.**
| Model | MMBench_DEV_Full (%) | MMvet | LLaVABench |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Hard ($\mathcal{V}$1+$\mathcal{L}$4) | vanilla | Easy ($\mathcal{V}$2+$\mathcal{L}$3) | Hard ($\mathcal{V}$1+$\mathcal{L}$4) | vanilla | Easy ($\mathcal{V}$2+$\mathcal{L}$3) | Hard ($\mathcal{V}$1+$\mathcal{L}$4) | vanilla | Easy ($\mathcal{V}$2+$\mathcal{L}$3) |  |
| DeepSeek-VL | 73.82 | 78.30 | 83.06 | 22.88 | 41.88 | 64.03 | 40.20 | 58.80 | 66.00 |
| Monkey-Chat | 73.06 | 77.45 | 82.65 | 29.72 | 40.96 | 40.96 | 38.70 | 48.50 | 73.00 |
| LLaVA-NeXT-Vicuna | 71.00 | 76.41 | 82.83 | 24.22 | 47.56 | 63.02 | 35.50 | 55.20 | 69.00 |
| XComposer2 | 76.94 | 83.02 | 87.08 | 30.55 | 38.48 | 60.55 | 42.30 | 58.00 | 73.50 |
| Yi-VL-34B | 71.40 | 77.91 | 85.30 | 22.15 | 30.55 | 60.22 | 37.50 | 44.30 | 59.70 |
| InternVL-2 | 78.00 | 86.78 | 88.17 | 36.33 | 54.49 | 68.15 | 36.00 | 53.00 | 68.30 |
| GPT-4o | 79.55 | 85.27 | 86.32 | 43.66 | 73.94 | 77.15 | 55.20 | 76.00 | 78.80 |
| Claude3-5V-Sonnet | 69.04 | 82.36 | 85.17 | 26.10 | 70.77 | 75.91 | 36.80 | 70.50 | 75.8 |

### 5.3 Result of compositional bootstrapping strategy

The performance of paired multimodal composition. We combine our strategies on $\mathcal{V}$ and $\mathcal{L}$ and obtained 12 dynamic variants that integrate visual and linguistic bootstrapping. Figure 5 shows the composition results in MMBench across three LVLMs, where each axis is organized from easy to difficult based on the complexity levels obtained from the above single-strategy experiments.
We observe that these LVLMs exhibit varying performance when faced with different variants, with overall trends being relatively consistent (*i.e.*, performance gradually decreases from the lower left to the upper right corner). Specifically, the combination of $\mathcal{V}1+\mathcal{L}4$ presents hardest challenges for LVLMs, while $\mathcal{V}2+\mathcal{L}3$ significantly boosts performance. These findings align with our observations from applying single-modal strategies. Results of paired multimodal composition on more benchmarks and LVLMs can be seen in Appendix figure 12.

Based on the above result, we extend these two extreme compositions (*i.e.,* $\mathcal{V}1+\mathcal{L}4$ and $\mathcal{V}2+\mathcal{L}3$ ) to the full set of MMBench, MMvet, and LLaVABench, to demonstrate the universality of our dynamic evaluation strategies. As displayed in Table 3, our VLB could be effective in evaluating the upper and lower bounds for the models’ performance. This capability to transform an original sample into versions with varying complexity not only tests but also enhances our understanding of the comprehensive capabilities of LVLMs. Through VLB, we achieve a more thorough evaluation, highlighting the robustness and adaptability of LVLMs to handle diverse and complex multimodal interactions.

The performance of multi-strategy composition.
Due to every single VLB strategy for image and question being atomic, we can explore the effect of different compositions according to the number of applied strategy: vanilla, single-strategy ($\mathcal{V}_{1}$), dual-strategy($\mathcal{V}_{1}$+$\mathcal{V}_{3}$), and tri-strategy ($\mathcal{V}_{1}$+$\mathcal{V}_{3}$+$\mathcal{L}_{4}$). Together with the easiest composition ($\mathcal{V}_{2}$+$\mathcal{L}_{3}$), Figure 6 demonstrates the performance of three advanced LVLMs on SEEDBench and MMvet. The lines show that as the number of hard strategies we use increases from 0-3, the difficulty of the benchmark also increases, leading to a gradual decline in the accuracy of LVLM.

### 5.4 Ablation Study and analysis

The effect of outpainting ratio in $\mathcal{V}_{3}$.
Since different expanding ratios can result in varied visual content, we investigated the influence of different expanding ratios ($1.25\times$, $1.50\times$, $1.75\times$, and $2\times$ ) in our $\mathcal{V}3$ strategy. The results in Figure 7 indicate that as the expanding ratios increase, the accuracy of the models in correctly answering questions decreases, and the rate of this decrease becomes steeper. This can be attributed to the fact that a larger expanding ratio introduces a more prominent and possibly distracting background into the image, whose visual information may not be directly relevant to the question, thereby diverting the LVLM’s attention from the crucial cues.

Can our VLB reduce data contamination?
Our dynamic strategy mitigates data contamination in two ways.
Firstly, since the new images and texts are generated using image editing tools and GPT-4V, where the process is random and variable, it is difficult for models to take the vast array of random variants into training.
Secondly, the newly generated sample differs from the contaminated original sample, thereby reducing the similarity with training data.
We applied the same methods of Section 3 to redetect the hardest variant in Figure 8 and found a significant reduction in data contamination rate among training sets.

Figure: Figure 6: The accuracy of LVLMs towards different multi-strategy compositions on SEEDBench.
Refer to caption: x6.png

Figure: Figure 7: The average accuracy of all evaluated LVLMs on different ratios of outpainting scales.
Refer to caption: x7.png

The effect of VLB on different tasks. Since our dynamic strategies are applicable to a wide range of multimodal tasks, we utilize SEEDBench to demonstrate the performance changes of LVLMs across various tasks. Figure 9 presents a comparison of average performance among the vanilla benchmarks, and two dynamic variants with $\mathcal{V}2+\mathcal{L}3$ and $\mathcal{V}1+\mathcal{L}4$ strategies. It is evident that performance changes more significantly in tasks like ‘Instance Interaction’, ‘Text Understanding’ and ‘Spatial Relation’. We found that images in ‘Instance Interaction’ and ‘Spatial Relation’ contain fewer objects, making the addition of new objects particularly disruptive to the LVLMs’ focus on core visual elements. For ‘Text Understanding’, it aligns with our strategic design hypothesis that introducing noise contexts can distract the models from understanding essential texts. Detailed error cases of these tasks on GPT-4o are in Appendix A.6.

Figure: Figure 8: Reduction of data contamination rate between the dynamic variant $d$ and the vanilla static benchmark $s$.
Refer to caption: x8.png

Figure: Figure 9: The accuracy for the hard variant, easy variant, and the vanilla static benchmark across different multimodal tasks.
Refer to caption: x9.png

## 6 Conclusion and Discussion

In this paper, we begin by quantitatively detecting the degree of contamination present in current benchmarks, indicating the need for dynamic multimodal evaluation (DME). And then we develop the first DME protocol with flexible complexity via Vision-Language Bootstrapping (VLB). VLB introduces bootstrapping strategies designed to transform original images and questions into new versions that simulate real user interactions in LVLMs, focusing on aspects of visual attention and linguistic understanding. We also employ predefined rules and the judge to ensure the newly generated problem preserves the consistency with the vanilla. With VLB, we build a series of dynamic variants upon an existing benchmark with diverse complexities. By conducting extensive evaluations using VLB, we demonstrate the limitations of current LVLMs, highlighting areas for potential improvement of LVLM in user interaction and adaption.

## Appendix A Appendix

### A.1 Bootstrapping Strategy Instruction

In this section, we detail the instructions for GPT-4 during the bootstrapping strategy process in Table LABEL:tab:vlb-prompt. They are strategy $\mathcal{V}_{1}$, $\mathcal{V}_{2}$, $\mathcal{L}_{3}$ and $\mathcal{L}_{4}$, respectively.

In strategy $\mathcal{V}_{1}$, we designed add-instruction, to prompt GPT-4V to return a set of insertable objects. Each object is in the form of (object name, bounding box).

Since GPT-4V is relatively weak in fine-grained visual grounding, in strategy $\mathcal{V}_{2}$, we utilize semantic masks to facilitate the appropriate selection of objects for removal. Leveraging segmentation models, we obtain semantic masks $\mathbf{M}$ of all objects within an image. Upon identifying semantic masks $\mathbf{M}$, each object is assigned a serial number. GPT-4V requires to pinoint the objects suitable for removal based on their numbers and names, each formatted as (object name, serial number).

In strategy $\mathcal{L}_{3}$, we seek to enhance the context with a caption focusing on information related to the question, generated by GPT-4V. Thus, we query GPT-4V to generate additional information that aids in responding.

In strategy $\mathcal{L}_{4}$, we aims at introducing nosing context, which pertains to adding context that is related to the image but not directly pertinent to the question. For instance, an observer might initially notice another object or the overall atmosphere before focusing on a specific object about which the question is posed.

**Table 4: The detailed instructions for image and language bootstrapping strategies.**
| Aspect | Input | Instructions |
| --- | --- | --- |
| $\mathcal{V}_{1}(I)$ Image<br>Bootstrapping | $<E_{s}>$ | There is a question about the image (resolution:$H\times W$) Its question is: $<Q>$. Its correct answer choice is: $<A>$.<br>Now please add an object into this image, but keep the original answer of the question the same,<br>which means the added object can not change the original answer<br>and the position of the added object can not cover the visual answer information.<br>Note that the size of the added object should be larger.<br>Please give me randomly 10 objects can be added and give me the exact bounding box coordinates<br>of each object according to the original resolution I give you.<br>The box in the output is the coordinates of the upper left corner $(x_{min},y_{min})$ and lower right corner $(x_{max},y_{max})$ of the object.<br>The output format should only be a list.<br>Each item in the list must be a dict. Do not output any extra information! Just output a list. The example of output format is:<br>$[{\text{name}:\text{name of object1},\text{box}:[x_{min},y_{min},x_{max},y_{<br>max}]},{\text{name}:\text{name of object2},\text{box}:[x_{min},y_{min},x_{max}<br>,y_{max}]},\cdots]$. |
| $\mathcal{V}_{2}(I)$ Image<br>Bootstrapping | $<E_{s},M>$ | Based on the image I provided (image includes the mark and the mask for each object),<br>tell me which objects can be removed without changing the answer to the question: $<Q>$.<br>Please give me a list containing exactly 5 objects can be removed. Do not output any extra information.<br>Just output a list. The example of output format is:<br>$[{\text{object_mark}:\text{xxx},\text{object_name}:\text{xxx}},{\text{object<br>_mark}:\text{xxx},\text{object_name}:\text{xxx}},\cdots]$<br>where object_mark is the given mark of object, and object_name is the name of specific object(can be repeated without distinction).<br>Note that the removed object can not change the answer to the question.<br>In the order you think is most obvious and appropriate to remove. |
| $\mathcal{L}_{3}(I)$ Text<br>Bootstrapping | $<I,Q>$ | Please add context to the question: introducing context or details to the question<br>that are relevant but not directly helpful in answering it.<br>Refer to the accompanying image $<I>$ for context. Question: $<Q>$. |
| $\mathcal{L}_{4}(I)$ Text<br>Bootstrapping | $<I,Q>$ | Now, based on this image, I want you to provide a paragraph that is irrelevant to the question: $<Q>$.<br>The paragraph should not help in answering the question at all.<br>Be careful not to directly describe the image, as that would aid in answering the question.<br>Instead, you should focus on an unrelated object or element in the image $<I>$, and let your imagination run wild.<br>The more creative and off-topic, the better. Please note the paragraph should be no more than 100 words. |

### A.2 Judge Instruction

Although our dynamic pipeline adheres to strict criteria to maintain the consistency of the original intent and informational content, it still unintentionally alters the meaning. Therefore, we introduce the judge module to compare the original with the modified images and the original with the rephrased questions, ensuring the preservation of the essence and factual accuracy. Specifically, to minimize excessive perturbations, the judge module assesses the newly generated images and questions from both the visual and textual modalities, respectively. Details of instructions can be seen in the following Table 5.

**Table 5: The detailed instruction given to the judge module.**
| Aspect | Input | Instructions | Output |
| --- | --- | --- | --- |
| Image<br>Judgement | $<E_{s},\mathcal{V}_{1}(I)>$ | Image-1: $<I>$ Image-2: $<\mathcal{V}(I)>$. I added an object to the image carefully. Please assess whether the modifications in the images affect the answer to the corresponding question.<br>You will see two images: one is the original image (Image-1), and the other is the modified image (Image-2).<br>The question is as follows: $<Q>$ with its associated answer: $<A>$. Please carefully compare the two images<br>and judge whether the changes in the modified image (Image-2) would affect the correctness of the answer to the question<br>relative to the original image (Image-1). Answer with “Yes” or “No” only. | ‘Yes’ / ‘No’ |
| Image<br>Judgement | $<E_{s},\mathcal{V}_{2}(I)>$ | Image-1: $<I>$ Image-2: $<\mathcal{V}(I)>$. I removed an object from the image carefully. Please assess whether the modifications in the images affect the answer to the corresponding question.<br>You will see two images: one is the original image (Image-1), and the other is the modified image (Image-2).<br>The question is as follows: $<Q>$ with its associated answer: $<A>$. Please carefully compare the two images<br>and judge whether the changes in the modified image (Image-2) would affect the correctness of the answer to the question<br>relative to the original image (Image-1). Answer with “Yes” or “No” only. | ‘Yes’ / ‘No’ |
| Image<br>Judgement | $<E_{s},\mathcal{V}_{3}(I)>$ | Image-1: $<I>$ Image-2: $<\mathcal{V}(I)>$. I expanded the image carefully. Please assess whether the modifications in the images affect the answer to the corresponding question.<br>You will see two images: one is the original image (Image-1), and the other is the modified image (Image-2).<br>The question is as follows: $<Q>$ with its associated answer: $<A>$. Please carefully compare the two images<br>and judge whether the changes in the modified image (Image-2) would affect the correctness of the answer to the question<br>relative to the original image (Image-1). Answer with “Yes” or “No” only. | ‘Yes’ / ‘No’ |
| Text<br>Judgement | $<E_{s},\mathcal{L}_{1}(Q)>$ | The two provided questions are semantically same and only have some minor differences. Question1: $<Q>$ Question2: $<\mathcal{L}_{1}(Q)>$.<br>The Ground Truth answer is: $<A>$. Please determine whether the ground truth answer applies to both Question 1 and Question 2? Only provide a simple output: Yes or No. | ‘Yes’ / ‘No’ |
| Text<br>Judgement | $<E_{s},\mathcal{L}_{2}(Q)>$ | Image: $<I>$ Given an image along with its corresponding question and answer, the original question has been modified into a semantically similar one.<br>The original question is $<Q>$, with its associated answer: $<A>$, and the modified question is $<\mathcal{L}_{2}(Q)>$.<br>Please determine whether the modified question changes the semantics of the original question, thereby potentially affecting the correctness of the original answer.<br>If it does change, output “Yes” ; if it does not change, output “No”. Please provide a simple output without explanation. | ‘Yes’ / ‘No’ |
| Text<br>Judgement | $<E_{s},\mathcal{L}_{3}(Q)>$ | Image: $<I>$ There is a question-answer pair regarding this image. Under the premise of not changing the semantics of the question,<br>I added some image descriptions into the original question, and got a modified question. The original question is: $<Q>$, the modified question is: $\mathcal{L}_{3}(Q)>$ and the original answer is: $<A>$.<br>Based on these information, do the modified question and the original question ask the same thing? Namely, are they both asking $<Q>$?<br>If they are, output “Yes” and if they are not asking the same thing, output “No”. Only provide me with a simple output without explanation. | ‘Yes’ / ‘No’ |
| Text<br>Judgement | $<E_{s},\mathcal{L}_{4}(Q))>$ | Image: $<I>$ There is a question-answer pair regarding this image. Under the premise of not changing the semantics of the question,<br>I added some image descriptions into the original question, and got a modified question. The original question is: $<Q>$, the modified question is: $\mathcal{L}_{3}(Q)>$ and the original answer is: $<A>$.<br>Based on these information, do the modified question and the original question ask the same thing? Namely, are they both asking $<Q>$?<br>If they are, output “Yes” and if they are not asking the same thing, output “No”. Only provide me with a simple output without explanation. | ‘Yes’ / ‘No’ |

### A.3 More cases of image bootstrapping

We provide some cases transformed by our image bootstrapping strategies in Figure 10. Visually, these dynamic variants show consistency with original images, ensuring the preservation of the essence. However, for LVLMs, the scenario is quite different. Despite the visual similarity, these modified images present new challenges.

Figure: Figure 10: More cases of image bootstrapping on SEEDBench, MMBench, and MME
Refer to caption: x10.png

### A.4 More cases of language bootstrapping

We present some results of bootstrapping the textual modality of existing benchmarks in Table 6.

**Table 6: More cases of language bootstrapping on SEEDBench, MMBench, and MME.**
| Benchmark | Strategy | Question |
| --- | --- | --- |
|  | Original | What type of flooring does the kitchen have? |
|  | $\mathcal{L}_{1}(Q)$ | What type of flooring does the cookroom have? |
| SEEDBench | $\mathcal{L}_{2}(Q)$ | The kitchen is equipped with what kind of flooring? |
|  | $\mathcal{L}_{3}(Q)$ | A stylish and modern kitchen interior design. There is a large window above the sink that allows ample natural light to flood the space.<br>The countertops appear to be a solid surface, possibly marble or quartz. What type of flooring does the kitchen have? |
|  | $\mathcal{L}_{4}(Q)$ | On the kitchen cupboard, there is a cookbook invitating people to a baking challenge that spreads the joie de vivre of pies.<br>This cookbook transforms an everyday kitchen into a nexus of community, where the artful dance of flour and sugar ignites a passion for the pastry arts amongst friends.<br>What type of flooring does the kitchen have? |
|  | Original | What type of environment is depicted in the picture? |
|  | $\mathcal{L}_{1}(Q)$ | What type of surroundings is depicted in the picture? |
| MMBench | $\mathcal{L}_{2}(Q)$ | What sort of setting does the picture portray? |
|  | $\mathcal{L}_{3}(Q)$ | The image portrays a modern, minimalist living space featuring an open living and dining area.<br>The room incorporates a neutral color palette with soft earthy tones and ample natural light.<br>The furniture and decor are a blend of natural materials like wood and woven fibers, contributing to a cozy and clean aesthetic.<br>The presence of several houseplants brings a touch of nature indoors, emphasizing a fresh and inviting atmosphere.<br>The environment is carefully organized and designed to create a harmonious and tranquil living space.<br>Given the details in the image, the depicted environment can best be described as a modern, minimalist, and cozy indoor living space.<br>What type of environment is depicted in the picture? |
|  | $\mathcal{L}_{4}(Q)$ | Imagine if instead of inorganic matter, all furniture was bioengineered, possessing a life cycle akin to plants or animals.<br>Consider a sofa that sprouts small, cushiony shoots each spring, which over the summer gradually bloom into lush, velvety surfaces perfect for lounging.<br>As autumn approaches, the colors of the cushions would transform into a spectacular display of reds, golds, and oranges,<br>before gently shedding layers, not dissimilar to deciduous trees.<br>The cycle would then culminate in a period of dormancy during winter, when the sofa regathers its strength to ensure it can unfurl its comfy resplendence<br>when warmth returns. What type of environment is depicted in the picture? |
|  | Original | Is there a baseball bat in this image? Please answer yes or no. |
|  | $\mathcal{L}_{1}(Q)$ | Is there a baseball bat in this figure? Please answer yes or no. |
| MME | $\mathcal{L}_{2}(Q)$ | Does there exist baseball bat in this image? Please give me an answer just yes or no. |
|  | $\mathcal{L}_{3}(Q)$ | The image depicts a scene from a baseball game. The batter is standing in the batter’s box,<br>poised to swing, while the catcher is crouched behind the home plate, ready to catch the ball.<br>An umpire stands behind the catcher, observing the scene closely. The dugout in the background is populated with team members watching the game.<br>Question: Is there a baseball bat in this image? Please answer yes or no. |
|  | $\mathcal{L}_{4}(Q)$ | In the world of sporting attire, the evolution of the catcher’s mitt is particularly fascinating.<br>This pivotal piece of equipment, cradled in traditions as old as the game itself, has undergone a considerable transformation over the years.<br>Initially just leather pads to protect the hands, modern catcher’s mitts are now engineered marvels featuring specialized cushioning,<br>tailored fit, and durable materials that withstand repetitive high-impact catches.<br>These mitts not only protect the player’s hands but also enhance their ability to catch fast-moving balls<br>with precision, showcasing a perfect blend of heritage and innovation in sports technology.<br>Question: Is there a baseball bat in this image? Please answer yes or no. |

### A.5 The effectiveness of judge module.

Although our newly transformed images and questions have passed an adversarial judged, for the sake of rigor, we further conduct a human verification to ensure that our variants maintain consistent with the original benchmark both visually and linguistically.
For each dynamic strategy, we recruit 20 human experts (with bachelor or higher degree) and randomly selected 100 samples from each strategy for each benchmark, totally 2100 samples. They are also tasked with judging the integrity and appropriateness of the modifications.
For visual dynamic strategy, we compose the original and edited image, with the corresponding question and answer, into the questionnaire. The experts are required to verify: (1) Whether the edited image maintains the same answer as the original image.
For linguistics dynamic strategy, we compose the original and transformed question, with the corresponding image and answer, into the questionnaire. And ask (2) whether the original and paraphrased questions were equivalent.
The evaluation required a simple ‘Yes’ or ‘No’ answer for our asked questions (1) and (2).
The positive results are shown in Figure 11, our human verification obtain a high accuracy rate, average of 96% for image dynamic strategy and 97% for question dynamic strategy on all three datasets. The result showcases a high level of alignment between the judge model and human verification, proving the equivalence and correctness of our dynamic methodology.

Figure: Figure 11: Human verification rate on each bootstrapping variant maintaining consistency with the vanilla.
Refer to caption: x11.png

Figure: Figure 12: Results of composition between image bootstrapping $\mathcal{V}$ and language bootstrapping $\mathcal{L}$ strategy.
Refer to caption: x12.png

### A.6 Analysis of error case

We visualize the GPT4o’s responses to images before and after dynamic changes.
Figure 13(a) shows an example of instance identity task, where adding a sofa caused the LVLM to wrongly answer a question it correctly responded in the original. This may be due to the added red sofa, which is brightly colored and distracts the model’s attention.
Figure 13 (b) is an example of spatial relation task, where the removal of a cabinet made the relative positions of the fan and stage clearer, allowing the LVLM to correctly answer the question it had previously gotten wrong.

Figure: Figure 13: (a) After the $\mathcal{V}_{1}$ strategy, GPT-4 incorrectly answered an instance identification question it had previously answered correctly. (b) By $\mathcal{V}_{2}$ strategy, GPT-4 correctly answered a spatial relation question it had previously answered incorrectly.
Refer to caption: x13.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. For example in figure 3(a), how is the visual attention and linguistic understanding converted into V1, V2, L1, L2 etc.?
2. How is it verified that the generated questions are not found in the pretraining data? Does using the VLB method ensure that? This is not discussed in the analysis.
soundness: 3
strengths: 1. The paper identifies an important research direction: existing benchmarks are static and because of large-scale pretraining data, it is hard to verify is some test data has leaked into the pretraining or training data. This makes evaluation difficult and the paper seeks to develop a new paradigm for evaluation.
2. The idea of using insights from user interactions to inform the transformations V and L is interesting.  
3. The experiments are comprehensive.
summary: The paper seeks to change the static nature and data contamination of benchmarks for vision-language models.  The paper introduces VLB: vision-langauge bootstrapping that dynamically generates new visual question answering samples via bootstrapping.  The goal is for the evaluation protocol to evolve with VLM capabilities.  The paper finds that existing VLMs struggle on the new benchmark.
weaknesses: 1. The role of user interaction is not defined in detail.  See Q1.
2. Question rephrasing has been previously explored in several other works on VQA (eg. VQA Rephrasings dataset) or robustness work such as VQA-LOL, VQA-Subquestions and others. What is the overlap of the proposed work with those benchmarks?
3. The work focuses only on VQA but there are several tasks that VLMs can perform.  Can the framework also handle capabilities that have to be evaluated without VQA?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please check the weakness section above.
soundness: 3
strengths: 1. Thorough experiment results validate the dynamic evaluation protocol VLB. First, a judge model is introduced to ensure that the dynamic image-question pair is still consistent with the original answer. Second, human examination on 2,100 samples verifies that less than 5% samples would introduce inconsistency.

2. The composition of multiple strategies effectively reduces data contamination and enables a wide range of difficulty levels. VLB can serve as a more reliable evaluation protocol than the traditional static ones.

3. The new evaluation protocol can be readily combined with existing LVLM benchmarks.
summary: This work proposes a new evaluation protocol, Vision-Language Bootstrapping (VLB), for comprehensively evaluating large vision-language models (LVLMs) with reduced data contamination and dynamic difficulty. Existing benchmarks like MM-Bench are constructed from fixed image-question pairs, which are partially observed in the training procedure of LVLMs. In VLB, both the image and the question can be modified to change the difficulty and avoid answering by memorization, while preserving the consistency with the original answer. Different operations in changing the image or question result in a series of difficulty levels. Extensive experiments are conducted to show performance change with VLB, through which the work poses new challenges for LVLMs.
weaknesses: 1. The major concern lies in the performance variance. Even with the same image-question sample and the same bootstrapping strategy, different dynamic samples can be generated, due to the randomness in GPT-4V and PowerPaint. However, the experiments do not show the scale of this variance caused by randomness. If this variance is large, the performance metrics may be less reliable.

2. Although the human verification (Figure 11) shows high consistency for each bootstrapping strategy, it is unclear if the consistency remains with composition of multiple strategies (e.g., V1+V3+L4). The errors may accumulate, and more changes to the image-question pair tend to break the original consistency.

3. The bootstrapping strategies rely on GPT-4V and PowerPaint. If they are replaced/combined with models with similar functions, can similar observations remain?

4. Some minor suggestions that do not affect the rating:
    - The bootstrapping strategies may be reordered to reflect the difficulty level. For example, how about making "removing existing objects" V1 and "adding new objects" V3?
    - In Figure 3(c), L3 should be "add relevant context."

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: Here I also list down a few minor suggestions.

1. The naming order of the VLB strategies could be changed so that the easy ones be introduced first. For example, it would feels more natural to first introduce L3, so that readers can more easily tell that, from Table 2, L3 is the positive alternative with additional helpful cues and lead to mostly improvement performance. The same idea applies for V2 as in Table 1.
2. The term 'data contamination' is not a coined term. I sense the authors want to describe the phenomenon that pre-training data already entail the contents for the supposedly unknown test set cases. In fact, this already has had a name **data leakage** as in (Chen et al, 2024). Let's stick with the established terminology. But feel free to correct me if the authors believe the two have any nuanced difference.

Typo: Table reference missing at Line 707.

Update 11/23: Raising my assessment according to the authors' response.

After all, I find this work very intriguing although the contribution feels a bit lackluster. However, I am open for reassessment after learning more from the exchange with the authors in the rebuttal period.

Chen et al, 2024. Are We on the Right Way for Evaluating Large Vision-Language Models?
soundness: 3
strengths: The paper is a pleasant read and is easy to follow. The paper is written in a well structured way following a clear plot line. I particularly find the parts where the bootstrapping strategies are introduced well written, which greatly helps me understand this work on an intuitive level.
summary: This work introduces VLB, a benchmark generation strategy aiming to prevent the baseline models from being evaluated on contaminated data (data leakage in essence). VLB extends current LVLM benchmarks by employing bootstrapping strategies to create altered test cases in variable difficulties in a controllable manner. The experimental results using VLB-modified benchmarks show that data leakage is evidently prominent in existing practices, and VLB may help establish fairer baselines for LVLM evaluations.
weaknesses: I do have one particular concern regarding the veracity of the VLB-modified data.

VLB strategies such as V1 (editing in a new object in the image) and L4 (adding irrelevant context in to the text) modify the original test case in a controlled manner. **However, how do we verify if the original test cases have been loyally modified in the way we want?** So far, such veracity verification steps are only observed in Figure 11 using human verification on a small batch of sampled data. The authors should consider additional automated verification techniques, such as the shift in semantic scores/image-text alignment metrics before vs after applying VLB strategies. 

After all, we do not want to re-evaluate LVLMs on wrongly generated test cases, which undermine the entire effort of VLB to start with.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 1
questions: 1. In practice, how can you ensure that when generating a lot of new samples, changes in images and questions do not introduce unnecessary bias or semantic errors?
2. Are there any automated validation methods that can guarantee semantic consistency across all generated samples?
3. When the external tools, such as PowerPaint, SAM and GPT-4V, being relied upon have limitations, for example, sometimes there will be some artifacts after using PowerPaint to remove an object, how does the DME ensure the fairness and accuracy of the evaluation?
4. How to improve the interpretability of evaluation results?
5. How to assist researchers in better understanding the specific areas where the model underperforms?
6. Can you provide the specific VLMs mentioned in Table 1? For example, InternVL2-1B or InternVL2-2B or … InternVL2-76B.
7. In Figure 4 “image outpainting” part, the proportion of the floor in the image V3 is larger than in the Vanilla image, and the question is “What type of flooring does the kitchen have?”, so in this situation, can “image outpainting” be defined as a hard task?
8. Have you resized the image when using PowerPaint and GPT-4V?
soundness: 3
strengths: 1. The paper proposes a novel dynamic multimodal evaluation framework VLB, which has a a flexible complexity adjustment evaluation mechanism.
2. By editing images and modifying questions, a set of evaluation samples with different levels of complexity can be generated. This method can be used to probe the upper and lower bounds of VLMs' capabilities in certain tasks.
3. The VLB framework is highly versatile. It can be easily plugged in a lot of benchmarks.
summary: The paper introduces a dynamic multimodal evaluation strategy to avoid data contamination for benchmarks. It shows that current popular evaluation benchmarks exist image-only contamination and image-text contamination. Image bootstrapping and language bootstrapping method can be used separately or in combination to adjust the difficulty of the questions while keeping the answers unchanged. Several experiments on 3 benchmarks and 8 VLMs are provided to verify the author's point of view.
weaknesses: 1. Authors use many models, such as GPT-4V, PowerPaint, SAM, to insert and remove objects from original images and employ GPT-4 for language reconstruction. Each step requires consistency evaluation, which may increase the complexity of implementation and lead to more unexpected errors or inconsistencies.
2. Generating new visual and language samples takes a long time, especially during large-scale evaluations. It also requires significant computational resources to produce new samples.
3. The VLB framework relies on PowerPaint, SAM and GPT-4V to generate new visual and language samples.
4. Although DME can generate diverse samples, the performance of VLMs may decrease on these samples, and the specific reasons for this are not clear.
5. Lack visualization for error analyses.

Meta-review: This paper introduces a novel evaluation protocol, Vision-Language Bootstrapping (VLB), designed to comprehensively assess large vision-language models (LVLMs) while minimizing data contamination and enabling dynamic difficulty adjustment. Unlike existing benchmarks,  VLB allows both the image and the question to be modified. These modifications adjust the difficulty level and prevent models from relying on memorization, while maintaining consistency with the original answer. By applying various operations to alter the image or question, VLB generates a spectrum of difficulty levels. Extensive experimental results demonstrate the impact of these adjustments, presenting new challenges for the evaluation of LVLMs. The draft got positive reviews from all reviewers and the AC agrees and recommends an accept.

Decision: Accept (Oral)