MM-R$^3$: On (In-)Consistency of Vision-Language Models (VLMs)
Authors: Shih-Han Chou, Shivam Chandhok, Jim Little, Leonid Sigal
Accepted at: acl25

MM-R^3: On (In-)Consistency of Vision-Language Models (VLMs)

Shih-Han Chou^{1,2}, Shivam Chandhok^{1,2}, James J. Little^{1}, Leonid Sigal^{1,2,3}
^{1}Department of Computer Science, University of British Columbia
^{2}Vector Institute for AI ^{3}Canada CIFAR AI Chair
{shchou75, chshivam, little, lsigal}@cs.ubc.ca

Figure 1: (Left) Overall results of VLMs on the MM-R^3 Benchmark. (Mid) Consider answering the three semantically identical questions for the top image and a given visual abductive reasoning question for the bottom images from the proposed MM-R^3 Benchmark. Humans are accurate and consistent in these tasks while VLMs are much less so. (Right) Results with the proposed adapter.

Abstract

With the advent of LLMs and variants, a flurry of research has emerged, analyzing the performance of such models across an array of tasks. While most studies focus on evaluating the capabilities of state-of-the-art (SoTA) Vision Language Models (VLMs) through task accuracy (\emph{e.g.}, visual question answering, grounding), our work explores the related but complementary aspect of \emph{consistency} – the ability of a VLM to produce semantically similar or identical responses to semantically similar queries. We note that consistency is a fundamental prerequisite (necessary but not sufficient condition) for robustness and trust in VLMs. Armed with this perspective, we propose the \emph{MM-R^3 benchmark}, which allows us to analyze performance, in terms of consistency and accuracy, of SoTA VLMs on three tasks: Question Rephrasing, Image Restyling, and Context Reasoning. Our analysis reveals that consistency does not always align with accuracy, indicating that models with higher accuracy are not necessarily more consistent, and vice versa. Furthermore, we propose a simple yet effective mitigation strategy in the form of an adapter module trained to minimize inconsistency across prompts. With our proposed strategy, we are able to achieve absolute improvements of 5.7% and 12.5%, on average on widely used VLMs such as BLIP-2 and LLaVa 1.5M in terms of consistency over their existing counterparts.

1 Introduction

Vision Language Models (VLMs) (Liu et al., 2023a; Li et al., 2023b; OpenAI, 2023; Xue et al., 2024), following and often built on top of purely lingual LLM (Brown et al., 2020; Touvron et al., 2023), have recently emerged as incredible tools for a broad range of visual understanding tasks, spanning captioning (Lin et al., 2014; Sharma et al., 2018; Chen et al., 2015), language grounding (Yu et al., 2016; Kazemzadeh et al., 2014; Liu et al., 2019), visual question answering (VQA) (Antol et al., 2015), and many others. As the number of such models and their capabilities explode, the research community is progressively focusing on benchmarking their capabilities by developing new benchmarks and testing harnesses.

Notable examples include MM-Bench (Liu et al., 2023c), SEED-Bench (Li et al., 2023a), MM-Vet (Yu et al., 2023), and others that define numerous tasks that capture a broad range of capabilities of such models (\emph{e.g.}, instance counting (Fu et al., 2023), spatial relation understanding (Yu et al., 2023), abductive (Hessel et al., 2022) and deductive (Park et al., 2020) reasoning). These benchmarks continue to shed light on abilities and limitations of VLMs by analyzing their \emph{accuracy}.

However, despite significant progress in the anal-
yses of VLMs, consistency1 – ability to produce semantically equivalent response with the semantically equivalent inputs, remains broadly overlooked. This is a fundamental requirement for VLMs to be reliably deployable for most tasks.

Anecdotally, LLMs and, by extension, VLMs are sensitive to their prompts which led to the widespread practice of prompt engineering. This is problematic as the models’ outputs may vary with the phrasing of a query rather than its actual intent, which undermines their reliability. Consider the example illustrated in Figure 1 (Mid) top: Most humans would realize that while the three questions (i.e R1, R2, and R3) are superficially different, the semantic meaning is the same. Hence even when the correct answer may not perhaps be apparent (i.e., “to be visible”), the same (consistent) answer should be produced. In contrast, asking models like BLIP-2 (Li et al., 2023b) to answer these questions results in varied responses “to protect them from splinters”,“to protect the horse’s legs”,“to make the hooves more visible” for the three questions considered. Similarly, it is obvious to humans that the object being masked in Figure 1 (Mid) bottom is the same irrespective of the type of the mask, and that the object in question is a “bench”. However, Qwen-VL-Chat’s responses vary, indicating “a bench”,“a yellow object”, “a green wooden slat sign” for different masks, highlighting the inconsistency in visual modifications.

It may be tempting to equate accuracy and consistency, but the relationship is more intricate. While it is true that for objective visual tasks (e.g., what color is an object), high accuracy will result in high(er) consistency, current VLMs are far from this high accuracy regime. Further, for more subjective visual tasks (e.g., abductive reasoning), high accuracy which tends to be measured as being similar to one of the sets of answers, may not lead to high consistency. In general, one can think of consistency as a necessary but not sufficient property of an AI system and one should seek to maximize both consistency and accuracy.

In this work, we first present a comprehensive analysis of SoTA VLMs in terms of their consistency. We do so by developing MM-R3 benchmark atop of the VQA task, where we produce both lingual rephrasings of the original questions (by leveraging GPT-3.5) and visual rephrasings of the image (through stylization) and measure both accuracy and consistency of the produced responses. We find that SoTA VLMs while often quite competitive in accuracy can differ substantially in their consistency of responses. For example, mPLUG-Owl2 (Ye et al., 2024) is much more susceptible to inconsistency when image inputs are perturbed while MoE-LLaVa (Lin et al., 2024) is more consistent in the change of the visual domain than the lingual domain. In addition, we also define an abductive task of predicting the contents of the masked region, where we find models like BLIP-2 and LLaVa 1.5M (Liu et al., 2023a, 2024) are lower in accuracy but have higher consistency. Overall results for VLMs are illustrated in Figure 1 (Left). We believe these findings both benchmark the capabilities of existing models and outline future directions and developments in more consistent VLMs and pre-training objectives (e.g., see efforts in LLMs (Aggarwal et al., 2023; Chen et al., 2024a; Jang and Lukasiewicz, 2023)).

Toward the latter goal, we propose a simple adapter module based strategy that effectively improves consistency. The adapter is flexible and can be added to any existing VLM. It sits between the VLM embedding layer and the frozen LLM decoder. The goal of the adapter is to modify the LLM’s embeddings such that they are invariant to surface form variations in the language prompt / question or the image. We test the proposed adapter in widely used VLMs such as BLIP-2 and LLaVa 1.5M models. The experiments show that it is able to improve performance on all three tasks in our proposed benchmark (shown in Figure 1 (Right)).

Our contributions are summarized as follows:

• We propose a new consistency benchmark, MM-R3, that enables evaluation of VLM’s ability to produce consistent responses to a range of inputs which are identical semantically, but differ in surface form. MM-R3 includes three tasks, covering visual and lingual domains.
• We conduct detailed analyses of SoTA VLMs in accuracy and consistency on this benchmark, taking the first step towards consistency in VLMs, which is a fundamental requirement for VLMs to be reliably deployable for most task.
• We propose a simple but effective adapter-based strategy that can be added to any VLM with lightweight training. Experiments on BLIP-2

1The definition that follows is one of strict consistency that we adopt in this paper. We note that such strict consistency may not be required of all VLMs or problems that they may be designed to solve. However, it is a useful a clean definition that we adopt here.
and LLaVa 1.5M models show that adding the adapter significantly improves performance on our benchmark regarding consistency metrics.

2 Related Work

Vision Language Models (VLMs). The study and development of VLMs (Liu et al., 2024; Li et al., 2023b; Bai et al., 2023b; OpenAI, 2023; Lin et al., 2024; Chen et al., 2023; Wang et al., 2023; Sun et al., 2024; Xue et al., 2024) has recently seen a surge in popularity. Motivated by the impressive achievement made by recent LLMs (Brown et al., 2020; Chowdhery et al., 2023; OpenAI, 2023; Touvron et al., 2023; Chung et al., 2024), researchers have ventured into augmenting these models with visual capabilities to tackle multimodal tasks more effectively. A pioneering effort in this realm was the Flamingo model (Alayrac et al., 2022), which integrated the CLIP image encoder with traditional LLMs. This initiative paved the way for the development of VLMs aimed at enhancing multimodal integration. For example, open-source models like LLaVa 1.5M (Liu et al., 2023a, 2024), BLIP-2 (Li et al., 2023b), Qwen-VL (Bai et al., 2023b), mPlug-Owl2 (Ye et al., 2024), MoE-LLaVa (Lin et al., 2024), and BLIP-3 (Xue et al., 2024). Meanwhile, proprietary models like GPT-4V (OpenAI, 2023), Gemini (Team et al., 2023), and GPT-4o (OpenAI, 2024) have demonstrated state-of-the-art performance, pushing the boundaries of research in this field. However, as VLMs become increasingly powerful, ensuring their reliability across both visual and linguistic domains poses a significant challenge. In this work, we aim to establish a new benchmark for evaluating the consistency of VLMs.

Vision-Language Benchmarks. Traditional Vision-Language (VL) benchmarks have predominantly centered on assessing performance in singular tasks, such as VQA (Antol et al., 2015; Goyal et al., 2017), OK-VQA (Marino et al., 2019), MSCOCO (Lin et al., 2014), and Visual Commonsense Reasoning (VCR) (Zellers et al., 2019). While these benchmarks are valuable, they fall short in fully gauging the comprehensive multimodal perception and reasoning capabilities of VLMs. In response to this gap, a new wave of VL benchmarks has been developed (Fu et al., 2023; Liu et al., 2023c; Yu et al., 2023; Li et al., 2023a; Zhang et al., 2024), tailored to the complex demands of VLMs. These benchmarks encompass a range of intricate multimodal tasks that necessitate a seamless integration of vision and language skills. For instance, MME (Fu et al., 2023) measures perceptual and cognitive skills across a total of 14 sub-tasks, offering a comprehensive measure of an VLM’s abilities. MME-RealWorld (Zhang et al., 2024) is the largest manually annotated benchmark to date, focusing on real-world applications with high-resolution images. Despite the advancements these benchmarks represent, their primary focus remains on measuring VLMs’ performance in terms of accuracy. MAD-Bench (Qian et al., 2024), on the other hand, explores VLM robustness by examining how models reconcile discrepancies between textual prompts and images. Our work takes a different stance by prioritizing the consistency of VLMs. In contrast to prior works, we propose a novel consistency benchmark that evaluates not just accuracy, but also the consistency of models across visual and linguistic domains.

Consistency in Language. Evaluating consistency in LLMs has emerged as a crucial area of inquiry, with semantic consistency being the most widely used focus in consistency analyses. This concept posits that a model should deliver consistent outcomes in semantically equivalent scenarios (Elazar et al., 2021). Notably, it has been observed that pre-trained language models may yield divergent predictions for masked language tasks when singular objects in queries are replaced with their plural counterparts (Ravichander et al., 2020) or queries are paraphrased (Elazar et al., 2021), indicating variability responses to semantically similar inputs. Building on these findings, a recent study (Jang and Lukasiewicz, 2023) comprehensively investigated LLM consistency. Besides consistency evaluation, to improve the robustness of LLMs, Liu et al. (2023b) proposes low-parameter finetuning methods that show a better out-of-distribution performance for generation and classification tasks. Similarly, Newman et al. (2022) proposes a P-Adapter that captures the factual information from the input prompts and improves the prediction consistency. Inspired by these pioneering efforts, our approach seeks to extend this evaluation to multimodal contexts. We employ three distinct tasks, question rephrasing, image restyling, and context reasoning, to analyze the consistency of responses generated by VLMs. Furthermore, we build an adapter to mitigate inconsistency that results from semantically equivalent lingual and visual prompts.
Table 1: Statistics of MM-R^3 Benchmark. We list the number of examples in each task as well as the source dataset from which we collect the examples. Test and Train splits are completely disjoint.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th colspan="2">#Examples</th>
    <th rowspan="2">Source</th>
  </tr>
  <tr>
    <th>Train</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>Question Rephrasing<br>Image Restyling<br>Context Reasoning</td>
    <td>16,894<br>27,226<br>30,003</td>
    <td>3,516<br>5,328<br>4,500</td>
    <td>InfographicsVQA (Mathew et al., 2022), OKVQA (Marino et al., 2019)<br>Google Landmarks v2 (Weyand et al., 2020), Indoor Scene (Quattoni and Torralba, 2009)<br>MSCOCO (Lin et al., 2014)</td>
  </tr>
</table>

3 MM-R^3 Benchmark

3.1 Overview of MM-R^3 Benchmark

We introduce a new benchmark designed to assess the semantic consistency of VLMs across both visual and linguistic domains. To achieve this, we have crafted three specific tasks: question rephrasing, image restyling, and context reasoning. These tasks are designed to probe the models’ ability to maintain consistency in their responses. Comprehensive statistics of our benchmark are presented in Table 1. The questions and images utilized in this benchmark are derived from existing datasets, which have been adapted to our tasks, ensuring a thorough evaluation of VLM consistency.

3.2 Tasks and Data Collection

We describe the tasks in MM-R^3 Benchmark and present two examples (see Figure 2) on each task.

Question Rephrasing. The goal of the question rephrasing task is to assess the ability of VLMs to provide consistent responses to variously rephrased questions. For this task, we employ GPT-3.5 to generate alternate versions of given questions. The questions and images are sourced from the InfographicsVQA (Mathew et al., 2022) and OKVQA (Marino et al., 2019) datasets. To make the rephrasing meaningful, we take the questions containing more than 10 words to do the rephrasing. Using the prompt “I have a question: (Question) Please give me three different types of rephrased questions to which the answer would be (Answer).”, we obtain three distinct rephrasings for each question. These rephrased questions, along with the corresponding images, are then presented to VLMs to evaluate their consistency with respect to linguistic surface form perturbations. We randomly select questions from the subset that meet the criteria. In total, we select 760 images and derive 3,516 rephrased questions.

Image Restyling. Humans possess remarkable ability to recognize places and objects in images despite changes in style. We posit that VLMs should demonstrate a similar level of adaptability. To this end, we have devised a task that assesses VLM consistency in the face of varied image styles. We choose stylization over more standard image augmentation transformations (e.g., TrivialAugment (Müller and Hutter, 2021) or RandAugment (Cubuk et al., 2020)) to ensure there is no data leakage; simpler data augmentations are often used in training VLMs themselves. To generate variations of styles, we leverage the style transfer model of (Johnson et al., 2016) along with Instance Normalization (Ulyanov et al., 2016). Our dataset comprises both outdoor scenes from the Google Landmarks Dataset v2 (Weyand et al., 2020) and indoor scenes from the Indoor Scene Dataset (Quattoni and Torralba, 2009). The images undergo transformations to exhibit four distinct styles: Candy, Mosaic, Udnie, and Grayscale. The VLMs are then tasked with describing the depicted places in two sentences by using the prompt “Please describe the place in the image in two sentences.”, based on these stylistically altered images. This approach allows us to evaluate VLMs consistency with respect to visual domain variations, e.g., the ability to describe the scene in a semantically similar manner irrespective of induced image style. We randomly collect 600 outdoor and 732 indoor scenes from the datasets, resulting in a total of 5,328 styled images.

Context Reasoning. Spatial context plays an important role in both human and computational perception (Torralba, 2003); it allows resoning under uncertainty. Testing VLMs on such a task carries a few key benefits. First, the task itself is clearly relevant and important. Second, it is non-deterministic and multiple pleasurable interpretations may rightfully exist. This is precisely a scenario where consistency maybe harder to achieve; helping to elucidate limitations of current models. To test VLMs in this setting, we introduce an image masking task where objects within images are randomly masked. We select images from the MSCOCO dataset (Lin et al., 2014), ensuring that the size of the masked object relative to the entire image
Figure 2: (a) The original question is transformed into three distinct versions, each employing a different questioning style. (b) The original images are converted into four stylistic versions. (c) In the Context Reasoning task, objects in the images are masked using various masking methods to test models’ inferential capabilities.

Original: What is the sum of all organisations who do not use social media while screening employees?
Rephrased 1: How many organizations do not currently use or have no intention of using social media for employee screening?
Rephrased 2: Do you include the total number of organizations that do not plan to incorporate social media into their employee screening process?
Rephrased 3: What is the combined count of organizations that do not utilize social media when screening potential employees?

Original: How old do you have to be in Canada to do this?
Rephrased 1: What is the minimum age requirement to participate in this activity in Canada?
Rephrased 2: At what age are individuals legally allowed to do this in Canada?
Rephrased 3: How old does one have to be in Canada to engage in this particular activity?

falls within a range of 0.1 to 0.25. This criterion guarantees that the masked objects are neither too small to be indiscernible nor too large to dominate the image. The masking is applied using three distinct types: lines, shapes, and colors. The choice of different types of masks is motivated by wanting to study to which extent the shape and color of a mask may bias accuracy and consistency of VLM responses. Specifically, we use [1, 3, 5, 7] lines, rectangle/ellipse shapes, and choose from red/blue/green/yellow/white/black colors for masking. Then, we present the masked images to VLMs alongside the query: “What kind of object is in the masked region?” In total, 1,500 images are randomly sampled from the subset that meet the constraint, resulting in 4,500 masked images.

Data quality validation. Since validating a large portion of the dataset (with 87,000 samples) manually would be exceedingly costly and time-consuming, we adopt two alternate strategies to evaluate the quality of our dataset: (1) We human validate 300 randomly sampled question-rephrasing pairs, and 300 stylized images. We find 93% language rephrasings and 86% image restylings have semantic equivalence for humans; (2) we use the InternVL-26B (Chen et al., 2024b) model (a strong VLM not part of our analysis, with capabilities exceeding GPT4-o in many cases) to automatically validate ALL of the data for the rephrasing task and find it to be 88% semantically equivalent according to InternVL. Note that this is likely a lower bound as InternVL itself is not perfect. This validates high quality of our data.

3.3 Evaluation Method

The responses expected from VLMs in our tasks are inherently open-ended, reflecting the diverse capabilities of these models. However, the design of our questions is meticulously aligned with the ground truth in the original dataset annotations. This alignment enables us to leverage the annotations effectively during evaluation. To systematically assess the performance of VLMs, we introduce four distinct evaluation methods, each capturing different aspects of model performance.

Accuracy (Acc). To evaluate accuracy we assess the responses from VLMs based on an exact match criterion (string matching) with the ground truth annotations. Specifically, if the ground truth annotation is encompassed within the VLM’s response, we consider the response correct. The accuracy score is then calculated as the average of correct responses across the benchmark dataset.

Similarity with GT (\( S_{GT} \)). Given the limitations of the exact match criterion, which may inaccurately categorize semantically similar responses as incorrect, we introduce a similarity metric designed to evaluate the nuanced semantic parallels between VLMs’ responses and the ground truth annotations. For instance, terms like person share semantic similarity with man and woman, yet would be deemed incorrect under a strict accuracy metric. To address this, our approach utilizes the semantic similarity metric, Sentence-Similarity (Reimers and Gurevych, 2019), which leverages LLM encodings to assess the similarity between responses and target answers. This metric provides a more
subtle assessment of model, rewarding semantic accuracy over mere lexical matching. The overall performance is the average score across the dataset.

Consistency Accuracy (Con). This metric is designed to quantify the proportion of responses that manifest a specified degree of semantic consistency. To achieve this, we leverage the same semantic similarity metrics to compute the pairwise similarity scores between responses, utilizing a threshold of 0.7 to delineate semantic consistency. The threshold is based on the observation of Semantic Textual Similarity benchmark (Cer et al., 2017). A response is deemed consistent if its similarity score with a paired response surpasses this threshold. The metric’s final value is obtained by averaging the proportion of consistent responses across the entire dataset, providing an aggregate measure of semantic consistency within the VLMs’ outputs.

Consistency Similarity (S_C). Similar to the Consistency Accuracy metric, we measure the consistency similarity by calculating pairwise similarity scores between responses. Instead of setting a threshold, we derive the final metric by computing the average of these similarity scores across the entire dataset. This allows us to more continuously assess the coherence of responses.

Overall Performance (O_all). Motivated by works in generalized few-shot recognition, we propose to use the Harmonic mean (\( H_{mean} \)) of correctness and consistency as a combined overall metric for model performance. We first calculate the average of Acc and S_GT, two metrics that evaluate correctness against the ground truth. Next, we compute the average of Con and S_C, two metrics assessing the consistency of generated responses. Finally, we combine these two averages into one single score using the harmonic mean, as we believe this approach can reduce bias when averaging values with large disparities. We use harmonic mean since ideally we want a model to be both correct and consistent and it helps balance these aspects:

\[
H_{mean}(mean(Acc, S_{GT}), mean(Con, S_C)).
\]

4 Evaluation Results

4.1 Models
All experiments are conducted on NVIDIA A40 GPUs. We evaluate a total of six widely used, open-sourced VLMs, including BLIP-2 (Li et al., 2023b), mPLUG-Owl2 (Ye et al., 2024), LLaVa 1.5M (Liu et al., 2024, 2023a), MoE-LLaVa (Lin et al., 2024), Qwen-VL-Chat (Bai et al., 2023b), and BLIP-3 (Xue et al., 2024) on our consistency benchmark. Additionally, the proprietary models, Gemini (Team et al., 2023), GPT-4V (OpenAI, 2023), and GPT-4o (OpenAI, 2024), are included to enrich our comparative analysis. Details of these models are provided in the Appendix A.3.

4.2 Main Results
The main results for different tasks are shown in Tables 2. For each task, we present results under Sampling and All Data. Sampling denotes querying the VLM multiple times with the identical input image and question to observe natural variation in responses without changing the conditions. This allows us to measure how much difference comes from the stochasticity of the model versus the rephrasing of the condition. For each example, we query the VLM four times and average across the whole dataset. All represents the mean performance across the dataset. Additional analysis in terms of consistency for correct vs. incorrect answers is given in Appx A.7.

Our key findings present intriguing insights. Notably, we observe a divergence between accuracy and consistency across the three tasks. While accuracy performance remains relatively similar and competitive among the different models, there is a marked disparity in their consistency performances. Moreover, we note significant variability in consistency across models, in contrast to the more stable accuracy performance. Furthermore, the variations in consistency are significantly more substantial in the image restyling and context reasoning tasks, suggesting that changes within the visual domain have a greater influence on consistency than linguistic modifications.

4.3 Detailed Analysis
Question Rephrasing. We provide quantitative results in Table 2. In the question rephrasing task, GPT-4V achieves the highest accuracy among the nine evaluated models, with Qwen-VL-Chat outperforming the rest of the open-source models. BLIP-2 and LLaVa 1.5M, set with a temperature of 0, achieve perfect consistency in sampling at 100%. However, their performance significantly drops when faced with rephrased questions, indicating a lower consistency in adapting to question variations compared to other models. It implies that BLIP-2 and LLaVa 1.5M might be sensitive to the input prompts. Another interesting observation is that, although Qwen-VL-Chat ranks lower
Table 2: Overall results on the MM-R^2 Benchmark. The numbers in red indicate the difference between Sampling and All results. The best-performing model in each metric is in bold and the second-best model is underlined.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">Sampling</th>
    <th colspan="6">All</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>GT</sub></th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>O<sub>att</sub></th>
    <th>Acc</th>
    <th>S<sub>GT</sub></th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>O<sub>att</sub></th>
  </tr>
  <tr>
    <td rowspan="7">Question Rephrasing</td>
    <td>BLIP-2</td>
    <td>30.29</td>
    <td>56.25</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>60.40</td>
    <td>25.13 (-5.16)</td>
    <td>52.91 (-3.34)</td>
    <td>48.15 (-5.18)</td>
    <td>63.90 (-36.10)</td>
    <td>46.00</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td>
    <td>40.10</td>
    <td>70.12</td>
    <td>58.58</td>
    <td>73.26</td>
    <td>60.03</td>
    <td>30.44 (-9.67)</td>
    <td>61.10 (-8.03)</td>
    <td>36.71 (-21.86)</td>
    <td>55.63 (-17.83)</td>
    <td>45.97</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td>
    <td>40.60</td>
    <td>70.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>71.35</td>
    <td>31.00 (-14.11)</td>
    <td>62.28 (-7.86)</td>
    <td>48.41 (-5.15)</td>
    <td>63.56 (-36.07)</td>
    <td>31.16</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td>
    <td>34.47</td>
    <td>65.04</td>
    <td>73.87</td>
    <td>84.88</td>
    <td>58.03</td>
    <td>28.85 (-5.26)</td>
    <td>59.97 (-7.93)</td>
    <td>42.72 (-13.28)</td>
    <td>61.16 (-32.72)</td>
    <td>37.82</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td>
    <td>52.05</td>
    <td>82.08</td>
    <td>93.25</td>
    <td>95.20</td>
    <td>78.56</td>
    <td>36.31 (-15.73)</td>
    <td><b>70.21</b> (<u>-1.87</u>)</td>
    <td>55.34 (-37.91)</td>
    <td>69.70 (-25.51)</td>
    <td>57.52</td>
  </tr>
  <tr>
    <td>BLIP-3</td>
    <td>32.70</td>
    <td>55.44</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>56.83</td>
    <td>30.94 (-1.76)</td>
    <td>60.98 (5.54)</td>
    <td>47.70 (-32.3)</td>
    <td>63.99 (-16.01)</td>
    <td>50.42</td>
  </tr>
  <tr>
    <td rowspan="3">Question Closed</td>
    <td>Gemini</td>
    <td>47.50</td>
    <td>71.98</td>
    <td>86.75</td>
    <td>90.48</td>
    <td>71.37</td>
    <td>39.35 (-8.15)</td>
    <td>66.22 (-5.36)</td>
    <td>58.26 (-28.49)</td>
    <td>70.66 (-19.82)</td>
    <td>58.04</td>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td>53.26</td>
    <td>64.75</td>
    <td>76.09</td>
    <td>81.58</td>
    <td>67.49</td>
    <td><b>50.22</b> (<u>-3.64</u>)</td>
    <td>67.27 (2.52)</td>
    <td>55.26 (20.83)</td>
    <td>69.18 (-12.41)</td>
    <td>60.43</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>57.06</td>
    <td>76.53</td>
    <td>85.42</td>
    <td>89.06</td>
    <td>75.66</td>
    <td>46.99 (10.07)</td>
    <td><b>69.04</b> (<u>+7.49</u>)</td>
    <td><b>60.87</b> (<u>+24.55</u>)</td>
    <td><b>72.01</b> (<u>+17.08</u>)</td>
    <td><b>61.94</b></td>
  </tr>
  <tr>
    <td rowspan="7">Image Restyling</td>
    <td>BLIP-2</td>
    <td>16.82</td>
    <td>16.40</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>28.49</td>
    <td>13.01 (-3.81)</td>
    <td>17.02 (0.62)</td>
    <td>38.36 (-61.64)</td>
    <td>62.83 (-37.17)</td>
    <td>23.15</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td>
    <td>15.71</td>
    <td>14.12</td>
    <td>53.47</td>
    <td>69.73</td>
    <td>24.01</td>
    <td>8.95 (-6.76)</td>
    <td>15.25 (1.13)</td>
    <td>26.29 (-27.18)</td>
    <td>59.21 (-10.52)</td>
    <td>18.86</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td>
    <td>15.24</td>
    <td>15.07</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>26.32</td>
    <td>10.47 (-4.77)</td>
    <td>15.49 (0.42)</td>
    <td>50.08 (-49.92)</td>
    <td>68.46 (-31.54)</td>
    <td>21.30</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td>
    <td>15.99</td>
    <td>15.48</td>
    <td>95.75</td>
    <td>86.36</td>
    <td>26.83</td>
    <td>12.93 (-3.06)</td>
    <td>16.60 (1.12)</td>
    <td>70.27 (25.48)</td>
    <td>74.79 (-11.57)</td>
    <td>24.54</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td>
    <td>13.81</td>
    <td>15.42</td>
    <td>77.28</td>
    <td>82.53</td>
    <td>24.71</td>
    <td>8.28 (-5.54)</td>
    <td>15.73 (0.31)</td>
    <td>23.10 (-54.18)</td>
    <td>53.29 (-29.24)</td>
    <td>18.26</td>
  </tr>
  <tr>
    <td>BLIP-3</td>
    <td>17.12</td>
    <td>15.08</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>27.73</td>
    <td>11.92 (-5.20)</td>
    <td><b>17.31</b> (<u>+2.33</u>)</td>
    <td>51.39 (-48.61)</td>
    <td>68.20 (-31.80)</td>
    <td>23.49</td>
  </tr>
  <tr>
    <td rowspan="3">Image Closed</td>
    <td>Gemini</td>
    <td>14.47</td>
    <td>16.55</td>
    <td>75.85</td>
    <td>75.88</td>
    <td>25.75</td>
    <td>13.68 (-0.79)</td>
    <td>16.15 (+0.4)</td>
    <td>53.37 (-22.48)</td>
    <td>68.95 (-6.93)</td>
    <td>23.98</td>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td>25.94</td>
    <td>15.34</td>
    <td>97.71</td>
    <td>85.07</td>
    <td>33.67</td>
    <td>9.72 (-16.22)</td>
    <td>15.90 (0.56)</td>
    <td>52.55 (-45.16)</td>
    <td>66.25 (-18.81)</td>
    <td>21.07</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>16.03</td>
    <td>17.65</td>
    <td>96.65</td>
    <td>84.20</td>
    <td>28.39</td>
    <td><b>16.31</b> (<u>+0.28</u>)</td>
    <td>16.83 (-0.82)</td>
    <td><b>79.60</b> (<u>+17.05</u>)</td>
    <td><b>77.07</b> (<u>+17.13</u>)</td>
    <td><b>27.38</b></td>
  </tr>
  <tr>
    <td rowspan="7">Context Reasoning</td>
    <td>BLIP-2</td>
    <td>28.20</td>
    <td>39.10</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>50.36</td>
    <td>27.91 (-0.29)</td>
    <td>38.97 (+0.13)</td>
    <td><b>82.44</b> (<u>+17.50</u>)</td>
    <td><b>88.80</b> (<u>+11.20</u>)</td>
    <td>48.10</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td>
    <td>24.43</td>
    <td>35.21</td>
    <td>33.36</td>
    <td>53.25</td>
    <td>35.32</td>
    <td>24.47 (0.03)</td>
    <td>35.34 (0.12)</td>
    <td>27.64 (-5.71)</td>
    <td>47.97 (-5.29)</td>
    <td>33.39</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td>
    <td>28.34</td>
    <td>42.54</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>52.34</td>
    <td>28.67 (0.32)</td>
    <td>42.52 (-0.03)</td>
    <td>68.04 (-31.96)</td>
    <td>77.02 (-22.98)</td>
    <td>47.75</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td>
    <td>26.13</td>
    <td>38.91</td>
    <td>75.49</td>
    <td>81.08</td>
    <td>45.95</td>
    <td>25.16 (-0.98)</td>
    <td>38.47 (-0.45)</td>
    <td>39.40 (-36.09)</td>
    <td>56.86 (-24.22)</td>
    <td>38.31</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td>
    <td>20.03</td>
    <td>32.30</td>
    <td>75.72</td>
    <td>83.19</td>
    <td>39.37</td>
    <td>20.11 (0.08)</td>
    <td>32.20 (-10.10)</td>
    <td>30.69 (-45.03)</td>
    <td>48.95 (-34.24)</td>
    <td>31.57</td>
  </tr>
  <tr>
    <td>BLIP-3</td>
    <td>28.00</td>
    <td>36.50</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>48.77</td>
    <td>27.96 (-0.04)</td>
    <td>36.48 (-0.02)</td>
    <td>40.02 (-50.88)</td>
    <td>54.88 (-45.12)</td>
    <td>38.38</td>
  </tr>
  <tr>
    <td rowspan="3">Context Closed</td>
    <td>Gemini</td>
    <td>55.60</td>
    <td>57.32</td>
    <td>68.08</td>
    <td>78.99</td>
    <td>63.88</td>
    <td><b>55.31</b> (<u>-0.29</u>)</td>
    <td><b>56.35</b> (<u>+0.97</u>)</td>
    <td>45.22 (-27.86)</td>
    <td>66.26 (-16.31)</td>
    <td>54.87</td>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td>33.72</td>
    <td>20.97</td>
    <td>37.21</td>
    <td>58.19</td>
    <td>34.76</td>
    <td>32.53 (-1.19)</td>
    <td>21.25 (-0.28)</td>
    <td>34.37 (-2.84)</td>
    <td>57.05 (-11.44)</td>
    <td>33.86</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>52.10</td>
    <td>31.82</td>
    <td>59.71</td>
    <td>70.50</td>
    <td>51.03</td>
    <td>51.73 (-0.37)</td>
    <td>31.67 (-0.15)</td>
    <td>49.49 (-10.22)</td>
    <td>66.82 (-8.38)</td>
    <td>48.57</td>
  </tr>
</table>

in accuracy compared to GPT-4V, it is better in consistency metrics.

Image Restyling. The results are shown in Table 2. In the image restyling task, GPT-4o stands out across three metrics and MoE-LLaVa leads among open-sourced models. Although BLIP-2 outperforms other models in accuracy, its consistency is exceptionally poor.

Context Reasoning. Table 2 presents the performance of various VLMs in the image masking task. Among the evaluated VLMs, the BLIP-2 and LLaVa 1.5M models achieve a better performance in consistency metrics. On the other hand, the Qwen-VL-Chat and BLIP-3 model show the weakest performance across the VLMs, with a notable drop in consistency metrics, which possibly means a lesser capability for abductive reasoning compared to its counterparts. The ground truth for this task is the annotations from the MSCOCO dataset (*e.g.*, “person”, “toilet”, “elephant”, “orange”, *etc.*). Since **Acc** represents the exact match between responses and the ground truth, we use **S<sub>GT</sub>** to complement the limitations of **Acc**. For example, if the ground truth is “person” and the response is “baseball player”, the **Acc** would be 0 since there is no exact match. However, the **S<sub>GT</sub>** would be 54.4, reflecting the semantic similarity between the two. There may indeed be ambiguities in the answers; however, given the size of the mask that correlates with object identity and surrounding context, humans are quite consistent on this task in most cases. We conducted a human study on 100 examples (300 images in total) for the context reasoning task. The scores for **Acc/S<sub>GT</sub>/Con/S<sub>C</sub>** are 66.0/82.0/95.0/97.4, which aligns with the hypothesis that humans are quite consistent.

Analysis Across Tasks. Across three tasks, Qwen-VL-Chat performs better in Question Rephrasing but falls short in Image Restyling and Context Reasoning among the open-sourced models. This disparity may come from its use of the state-of-the-art LLM, Qwen (Bai et al., 2023a), which likely provides Qwen-VL-Chat with superior initial language representations. On the other hand, BLIP-2 achieves the best performance in Context Reasoning. This is possibly due to its unique image-text learning during pre-training, which could facilitate a better contextual understanding of images. We believe the matching between image and language indeed helps the models learn the context in the image. Meanwhile, LLaVa 1.5M and MoE-LLaVa exhibit balanced performances across all
Figure 3: (Left) Model Overview. The adapter is added between the Vision-Language Encoding and Generation models. The encoding and decoding parts are frozen during the training of the adapter. (Right) Vision-Language Encoding details of BLIP-2 and LLaVa 1.5M.

tasks, achieving a good trade-off between accuracy and consistency. Among the closed-source models, GPT-4o outperforms Gemini and GPT-4V in all three tasks. However, a notable observation is that current top-performing VLMs still drop significantly on consistency when facing changes in visual and linguistic domains. The gaps suggest that further effort is needed to enhance VLMs’ performance in terms of consistency.

Qualitative Results. We show some qualitative results of the question rephrasing task in Figure 4. The closed-source models outperform other models in terms of both accuracy and consistency. While the accuracy of the evaluated VLMs may not meet high standards, the similarity between the generated responses and the ground truth maintains a satisfactory level (\( S_{GT} \)), suggesting that the responses are semantically aligned with the ground truth. Regarding consistency, the majority of VLMs demonstrate the capability to generate semantically similar answers to rephrased questions. More results for question rephrasing, image restyling, and context reasoning are in the Appendix A.13.

5 Consistency Improvement

The analyses on the MM-R^3 benchmark in Section 4 show that consistency drops significantly with semantically equivalent prompts. To overcome this issue, we propose a simple strategy based on an adapter module that can be plugged into any VLM with lightweight training (Figure 3 (green)). The goal of the adapter is to help VLMs overcome the variability of visual and language prompts by making them invariant to surface form variability in data and ensuring a single consistent output.

5.1 Approach

The design of the adapter is inspired by (Newman et al., 2022). The adapter model takes the embeddings of the input prompts and outputs a new sequence of continuous embeddings that are used as input to the LLM decoder for generation. Specifically, the adapter takes the outputs of the Vision-Language encoder, passes them through a Bi-LSTM module and a max-pooling procedure to obtain the global embeddings from the input tokens. Then, an MLP is added afterward to project the max-pooling output to the size of the prefix that will be added in front of the original embeddings. In this way, the adapter not only captures the semantic embeddings from the prompts but also retains all of the original embeddings for the LLM decoder. During fine-tuning, we update only the parameters of the adapter while freezing the vision-

![Model overview diagram showing the adapter between Vision-Language Encoding and Generation models](page_68_68_670_246.png)

![Vision-Language Encoding details diagram showing BLIP-2 and LLaVa 1.5M](page_748_68_670_246.png)

![Qualitative results table comparing question rephrasing accuracy and consistency for various models](page_68_393_670_246.png)

Figure 4: Qualitative Results for the Question Rephrasing Task. Q and A are the original question and answer. R1, R2, and R3 denote the rephrased questions.
Table 3: Improvement Results on Three Tasks. The consistency metrics on three tasks on both BLIP-2 and LLaVa 1.5M models significantly improved after adding the adapter.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Question Rephrasing</th>
    <th colspan="4">Image Restyling</th>
    <th colspan="4">Context Reasoning</th>
  </tr>
  <tr>
    <th>Acc</th><th>SCT</th><th>Con</th><th>Sc</th><th>O_{all}</th><th>Acc</th><th>SCT</th><th>Con</th><th>Sc</th><th>O_{all}</th><th>Acc</th><th>SCT</th><th>Con</th><th>Sc</th><th>O_{all}</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>25.1</td><td>52.9</td><td>48.2</td><td>63.9</td><td>46.0</td><td>13.0</td><td>17.0</td><td>38.4</td><td>62.8</td><td>23.1</td><td>27.9</td><td>39.0</td><td>82.4</td><td>88.8</td><td>48.1</td>
  </tr>
  <tr>
    <td>LLaVa</td><td>26.9</td><td>59.2</td><td>32.5</td><td>53.8</td><td>43.1</td><td>9.6</td><td>14.9</td><td>19.0</td><td>56.9</td><td>18.5</td><td>20.1</td><td>28.9</td><td>25.9</td><td>42.3</td><td>28.5</td>
  </tr>
  <tr>
    <td>Adapt</td><td>BLIP-2</td><td>24.1</td><td>62.1</td><td>58.0</td><td>71.2</td><td>51.7</td><td>27.0</td><td>27.6</td><td>49.1</td><td>66.7</td><td>37.1</td><td>53.5</td><td>51.8</td><td>88.5</td><td>94.0</td><td>66.8</td>
  </tr>
  <tr>
    <td>Adapt</td><td>LLaVa</td><td>28.7</td><td>64.4</td><td>42.5</td><td>61.1</td><td>49.0</td><td>20.6</td><td>25.9</td><td>32.7</td><td>52.1</td><td>30.0</td><td>55.3</td><td>71.9</td><td>62.2</td><td>74.8</td><td>66.0</td>
  </tr>
</table>

language encoder and the language decoder. As shown in Figure 3, the yellow parts are frozen, and the green one represents the training component.

5.2 Experiments

Training Data. We leverage the same data generation pipeline as the MM-R^3 benchmark to generate training data samples. Note that these training samples are completely disjoint with the samples in MM-R^3 benchmark. In total, there are 16, 894 rephrased question-image pairs for the Question Rephrasing Task, 27, 226 styled images for the Image Restyling Task, and 30, 003 masked images for the Context Reasoning Task for training.

Implementation Details. We select BLIP-2 and LLaVa 1.5M for consistency improvement experiments since they are widely used in recent works, have low consistency compared to other models across lingual changes and allow us to show efficacy of our approach on different types of VLMs families (i.e, use only CLIP vs Q-former based architectures). More details are given in Appx A.12.

Results. The performance with and without our adapter, on all three tasks, is illustrated in Table 3. In question rephrasing the accuracy of the model with the adapter is marginally better (improvement of +1.8 for LLaVa 1.5M), however, the consistency shows a very significant margin of improvement (+9.8 and +10 respectively). This is consistent with our earlier observation that accuracy and consistency are not necessarily aligned. The original VLMs are already well trained for VQA tasks and so large accuracy boost is not expected. However, they are lacking in consistency (see Section 4), hence the large improvement on those metrics. For image restyling and context reasoning we do see a large improvement on both the accuracy and consistency. This is largely because original VLMs are not trained on data of this form. Hence, the introduced adapter can both fine-tune performance on the new data and improve consistency on them at the same time; as shown in the Figure 4.

To validate our adapter has not degraded original VLM models, we evaluate on OKVQA dataset before and after the adapter-based fine-tuning. The Original LLaVa 1.5M model shows 58.04 in Acc and the finetune one is 57.12. This indicates the proposed adapter significantly improves consistency, with minor, if at all, impact on capabilities.

6 Conclusion

In this paper, we explore and analyze consistency of VLMs and its relationship to accuracy. We do so by introducing the MM-R^3 Benchmark, comprising three tasks – Question Rephrasing, Image Restyling, and Context Reasoning – to conduct a comprehensive analysis of SoTA VLMs in terms of both accuracy and consistency. Our analysis reveals that higher accuracy does not necessarily equate to greater consistency in models, and vice versa. In addition, we observe significant variations in the consistency of SoTA models, while their accuracy levels tend to be more stable across models. These findings suggest that future VLM development and objectives might benefit from a dual focus, emphasizing the optimization of consistency and the improvement of overall accuracy. Furthermore, we propose an effective adapter to improve consistency performance. Experiments on BLIP-2, LLaVa validate improvements.

Acknowledgments. This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chairs, NSERC Canada Research Chair (CRC), and NSERC Discovery Grants. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, the Digital Research Alliance of Canada\footnote{https://vectorinstitute.ai/\#partners}, companies sponsoring the Vector Institute, and Advanced Research Computing at the University of British Columbia. Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award.
7 Limitations

In this work, we focus on semantic input perturbations in both visual and linguistic modalities, as well as semantic output equivalence. Achieving and quantifying this is challenging. Our goal is to develop automatic metrics to simplify the evaluation process. However, the methods for evaluating consistency are not limited to the metrics we propose. For example, some studies use LLMs to assess whether responses are consistent, and future work could further explore this approach with more powerful LLMs.

Additionally, as defined in the introduction, we consider consistency to be a measure of average performance under semantic perturbations, though alternative definitions may also exist. We believe this is an important future direction, where datasets can be expanded and the definition of consistency refined to better encompass a wider range of scenarios.

References

Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023. Let’s sample step by step: Adaptive-consistency for efficient reasoning with llms. In EMNLP.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visual language model for few-shot learning. In NeurIPS.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In ICCV.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuncheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.

Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliaj Li, Thomas Unterthiner, and Andreas Veit. 2021. Understanding robustness of transformers for image classification. In ICCV.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.

Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). Ieee.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. International Workshop on Semantic Evaluation (SemEval-2017).

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghu Ramakrishnan, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.

Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Ke-fan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024a. Universal self-consistency for large language model generation. ICML Workshop on In-Context Learning.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek
Rao, Parker Barnes, Yi Tay, Noah Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaeck Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research (JMLR).

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research (JMLR).

Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML. PMLR.

Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, volume 33.

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Erratum: Measuring and improving consistency in pretrained language models. ACL.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR.

Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022. The abduction of sherlock holmes: A dataset for visual abductive reasoning. In ECCV.

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are features. NeurIPS.

Myeongjun Jang and Thomas Lukasiewicz. 2023. Consistency analysis of ChatGPT. In EMNLP.

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In AAAI.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referring game: Referring to objects in photographs of natural scenes. In EMNLP.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2023a. Seed-bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML.

Linjie Li, Zhe Gan, and Jingjing Liu. 2020. A closer look at the robustness of vision-and-language pre-trained models. ArXiv.

Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In CVPR.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. In NeurIPS.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys.

Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. 2019. Clevr-ref+: Diagnosing visual reasoning with referring expressions. In CVPR.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Yike Yuan, Wangbo Zhao, Ji-aiqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023c. MMBench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR.

Minesh Mathew, Viraj Bagal, Rubén Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In WACV.

Milad Moradi and Matthias Samwald. 2021. Evaluating the robustness of neural language models to input perturbations. ArXiv.

Samuel G. Müller and Frank Hutter. 2021. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In ICCV, pages 774–782.

Benjamin Newman, Prafulla Kumar Choubeey, and Nazneen Rajani. 2022. P-Adapters: Robustly extracting factual information from language models with diverse prompts. In ICLR.

OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.

OpenAI. 2024. GPT-4o. https://platform.openai.com/docs/models/gpt-4o.

Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. In ECCV.

Yusu Qian, Haotian Zhang, Yinfei Yang, and Zhe Gan. 2024. How easy is it to fool your multimodal llms? an empirical analysis on deceptive prompts. arXiv preprint arXiv:2402.13220.

Ariadna Quattoni and Antonio Torralba. 2009. Recognizing indoor scenes. In CVPR.

Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Joint Conference on Lexical and Computational Semantics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL.

Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiyng Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Generative multimodal models are in-context learners. In CVPR.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

A. Torralba. 2003. Contextual priming for object detection. In IJCV.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Qi Ji, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. CogVLM: Visual expert for pretrained language models. In NeurIPS.

Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. 2020. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In CVPR.

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, and Ran Xu. 2024. xGen-MM (BLIP-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration. In CVPR.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In ECCV.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. MM-Vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In CVPR.

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-off between robustness and accuracy. In ICML. PMLR.

Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. 2024. MME-RealWorld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257.

Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. 2023. Prompt-bench: Towards evaluating the robustness of large language models on adversarial prompts. ArXiv.
A Example Appendix

In the appendix, we present additional details and results to complement the main paper.

A.1 Adversarial robustness vs. consistency.

The adversarial robustness is related to but different from the consistency we study in this paper. First, most adversarial robustness approaches (Carlini and Wagner, 2017; Croce and Hein, 2020; Ilyas et al., 2019; Zhang et al., 2019; Bhojanapalli et al., 2021) operate in classification settings. Models (Jin et al., 2020; Zhu et al., 2023; Moradi and Samwald, 2021) operate on LLMs that have no entitlement of vision and language, and (Li et al., 2020) only studies CLIP (which is a particularly simple contrastive VLM variant). Notably, none of the models deal with VLMs with continuous text outputs. Second, they assume the presence of an adversary agent that attempts to find small, local, and often imperceptible, perturbations to inputs (e.g., (Bhojanapalli et al., 2021) propose pixel level noise perturbations for vision models; (Zhu et al., 2023) propose typos and synonyms for LLMs; (Moradi and Samwald, 2021) propose character and word level deletions, repetition, etc.), that “produce an incorrect response” (Zhu et al., 2023) or a “decrease in overall classification performance” (Li et al., 2020). In other words, robustness is closely tied to accuracy; i.e., robustness only makes sense in the context of a capable model, for samples that the original model is able to classify correctly. Third, adversarial robustness models, particularly those that attempt to provide theoretic guarantees, quantify worst-case performance under an adversary attack. In contrast, in studying consistency in VLMs, we focus on a broad class of VLM models that produce open-world textual outputs (including both open- and closed-sourced); this is well beyond CLIP discussed in (Li et al., 2020) (which is the closest among suggested citations). Also, we focus on semantic input perturbations (rephrasing and restyling) of both visual and lingual modalities and semantic output equivalence. This is much harder to achieve and quantify. This also goes significantly beyond local word/character perturbations in LLMs or pixel noise perturbations in vision robustness literature. Importantly, the notion of consistency is entirely devoid of the accuracy or correctness of the original model. Specifically, we study consistency for both all responses and specifically failure cases (see supplements). A model can be trivially consistent by always responding with the same phrase, irrespective of the input, however, such a model would not generally be considered either accurate or robust under most standard definitions of those two properties. Further, consistency does not assume an adversary, but rather a cooperative agent. In other words, the only perturbations we consider are those likely to be generated by a “typical” user (not one that tries to fool a model). Overall, consistency does not guarantee robustness. On the other hand, a robust model may also not necessarily guarantee consistency, because typical robustness measures ability for an adversary to flip the decision from correct to incorrect. In more complex tasks (e.g., VQA, captioning), there may be multiple correct answers and also many ways to be incorrect. Consistency measures semantic equivalency even within these classes, which robustness typically does not. Finally, consistency as we define it, is a measure of average performance under semantic perturbation, not one of worst-case performance.

A.2 Data Generations and Tasks Pipeline

We provide a comprehensive pipeline that details both the data collection process and the methods of prompting VLMs for each task, shown in Figure 5.

A.3 Details of VLMs

BLIP-2 (Li et al., 2023b): The BLIP-2 model addresses the modality gap through a novel lightweight Querying Transformer, the Q-former, leveraging a two-stage pre-training approach. Despite its lack of multimodal instruction tuning, BLIP-2 retains the LLM’s capacity for following instructions. For our evaluations, we employed the BLIP-2 FlanT5XXL in our evaluations.

mPLUG-Owl2 (Ye et al., 2024): mPLUG-Owl2 utilizes a modularized network design tailored for handling multi-modal inputs. It contains a modality-adaptive module to align different modalities into a shared semantic space for relational learning. The model’s architecture, including the visual encoder, visual abstractor, and language decoder, are all updated during training. We use mplug-owl2-llama2-7b for inference and our performance analysis.

LLaVa 1.5M (Liu et al., 2024, 2023a): LLaVa is an end-to-end model combining LLaMA/Vicuna on GPT-generated multimodal instruction-following data. It provides general-purpose visual and language understandings, boasting chat capabilities that parallel the multimodal proficiency of
Q: What is the name of the purple vegetable on the pizza?
A: Onion.

I have a question: Please give me three different types of rephrased questions to which the answer would be <A>.

GPT-4

Q1: What is the name of the round, white vegetable commonly used in cooking?
Q2: Can you name a commonly used vegetable that is often sauteed or grilled?
Q3: Do you know the name of the vegetable with layers and a strong, distinct flavor that makes people cry when they chop it?

Prompt: Please answer the question with a very-short answer. <Q1/Q2/Q3>.

MLLM

A1: onion
A2: onion
A3: onion

(a) Question Rephrasing

A: kindergarten

Neural-Style
Grayscale

candy   mosaic   udnie

Prompt: Please describe the place in the image in two sentences.

MLLM

A1: classroom
A2: kindergarten
A3: classroom
A4: kindergarten

(b) Image Restyling

A: birds

Masking

Lines: [1, 3, 5, 7]
Colors: [red, blue, green, yellow, white, black]
Mask types: [rectangle, ellipse]

Prompt: What kind of object is in the masked region?
A: birds

MLLM

A1: birds
A2: birds
A3: birds

(c) Context Reasoning

Figure 5: The pipeline for the data collection process and prompting VLMs for each task.

GPT-4. We conducted our experiments using the 11ava-v1.5-7b version.

MoE-LLaVa (Lin et al., 2024): The MoE-LLaVa model incorporates a Mixture of Experts (MoE) architecture with learnable routers, comprising multiple sparse paths that uniquely activate only the top-k experts during deployment while keeping the remaining experts inactive. This design allows MoE-LLaVa to achieve performance comparable to other high-end VLMs with the provided sparse path. We use MoE-LLaVA-Phi2-2.7B-4e-384 version for evaluation.

Qwen-VL-Chat (Bai et al., 2023b): Qwen-VL-Chat builds upon the foundation of Qwen-VL, with training that encompasses not just traditional image descriptions and question-answering tasks, but also extends to grounding and text-reading capabilities through the alignment of image-caption-box tuples. The version tested and analized in our experiments is Qwen-VL-Chat.

BLIP-3 (Xue et al., 2024): BLIP-3 (xGen-MM) consists of a Vision Transformer, a vision token sampler to downsample the image embeddings, and a pre-trained Large Language Model. BLIP-3 enables higher-resolution images as inputs by using patch-wise encoding. The patch-wise encoding preserves the resolution of the original images by encoding the split image patches separately. Then a perceiver resampler is used to downsample the visual tokens before sending them to the LLM. This design allows BLIP-3 to reduce the sequence length of vision tokens while keeping the higher-resolution images as inputs. We evaluate the Consistency Benchmark on xgen-mm-phi3-mini-instruct-r-v1 version.

Gemini (Team et al., 2023): Gemini is developed
by Google. It is a multimodal model using a Transformer to process various inputs, such as text, images, audio, and video. Unlike models using separate modules to encode different types of data, Gemini uses a shared Transformer. This enables the model to leverage shared reasoning capabilities. We use gemini-1.5-flash for evaluation.

GPT-4V (OpenAI, 2023): GPT-4V extends the capabilities of Generative Pre-trained Transformers by integrating visual understanding, enabling it to process and generate content based on both textual and visual inputs. The development represents a significant advancement in AI, making it a versatile tool for a wide range of applications that require the understanding of both text and imagery. We evaluate the Consistency Benchmark on gpt-4-vision-preview version.

GPT-4o (OpenAI, 2024): GPT-4o (“o” for “omni”) is the most advanced model released by OpenAI. It accepts multimodal inputs, e.g. texts and images. We leverage gpt-4o version to evaluate the Consistency Benchmark.

A.4 Similarity Metrics Details

For evaluating Similarity with Ground Truth (\( S_{GT} \)), Consistency Accuracy (Con), and Consistency Similarity (\( S_C \)), we leverage semantic similarity metrics, Sentence-Similarity (Reimers and Gurevych, 2019). This metric utilizes large language model encodings to compare the semantic content of predicted and target texts. Specifically, Sentence-Similarity achieves this by transforming input texts into embeddings via a pre-trained Transformer model, capturing their semantic differences. The similarity between these embeddings, and thus the texts, is calculated using cosine similarity, providing a measure of their semantic closeness.

A.5 Post-processing of Answers

In the Context Reasoning task, we notice that VLMs often repeat phrases from the question, such as “the masked region is...” which raises the consistency scores due to these repetitive terms when calculating answer similarity. However, our primary interest lies in identifying the objects behind the masks. To address this, we conduct post-processing on the responses before metric calculation. This involves filtering out common words like *there*, *is*, *of*, *in*, *the*, etc., and phrases frequently repeated from the question, such as *masked region* and *image*. This adjustment allows for a more fair comparison by focusing on the substance of the responses rather than their structural similarities.

A.6 Evaluation procedure

Calculation of Accuracy (Acc): To clarify the calculation of accuracy, we do case-insensitive substring matching to validate the response. This works because GT responses tend to be single words or short phrases. Consider the example with three different answers (“columbia”, “north face”, and “no”) and the ground truth answer is “north face”. Acc for three answers is 0/100/0. As a result, the average score for this example will be 33.3 as reported.

Calculation of similarity with GT (\( S_{GT} \)): As the exact match criterion has some limitations, i.e. it may inaccurately categorize semantically similar responses as incorrect, we use a similarity metric in the form of Sentence BERT embeddings.

Calculation of Consistency Accuracy (Con): We compute the pairwise similarity scores between responses using Sentence BERT and utilize a threshold of 0.7 to delineate semantic consistency. Consider again the example with three different answers (“columbia”, “north face”, and “no”), since none of these are semantically similar to one another, the pair-wise Sentence BERT scores are 0.27/0.14/0.24 — all below 0.7 threshold and resulting in Con of 0.

Calculation of Consistency Similarity (\( S_C \)): We compute the pairwise similarity scores between responses and average them. Using the same example above, the SC score will be \( (0.27 + 0.14 + 0.24)/3 = 0.21 \).

A.7 Failure Cases

In Table 4, *Failure Cases* means we only focus on incorrect answers under different changes.

A.8 Analysis on Model Size

Previous studies show that the number of parameters in VLMs affects performance in downstream tasks. For example, BLIP-2, the model achieves greater performance when the number of parameters is larger in VQA, Image Captioning, and Image-Text Retrieval (Li et al., 2023b). A similar trend is found in the LLaVA 1.5M model (Liu et al., 2024). As a result, we are curious how consistency is impacted with different sizes of models. We evaluate the BLIP-2 and LLaVA 1.5M models with small and large numbers of parameters. As shown in Table 5, large models outperform small models on *Acc* and *S_{GT}* in all three tasks. How-
Table 4: Failure results of different models.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Question Rephrasing Task</th>
    <th colspan="3">Image Restyling Task</th>
    <th colspan="3">Context Reasoning Task</th>
  </tr>
  <tr>
    <th>S<sub>GT</sub></th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>S<sub>GT</sub></th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>S<sub>GT</sub></th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td>BLIP-2</td>
    <td>46.44</td>
    <td>41.75</td>
    <td>59.33</td>
    <td>16.87</td>
    <td>37.15</td>
    <td>62.01</td>
    <td>28.68</td>
    <td>77.78</td>
    <td>86.41</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td>
    <td>54.58</td>
    <td>27.96</td>
    <td>49.25</td>
    <td>15.23</td>
    <td>25.46</td>
    <td>58.90</td>
    <td>31.05</td>
    <td>23.29</td>
    <td>44.81</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td>
    <td>56.26</td>
    <td>39.83</td>
    <td>57.99</td>
    <td>15.43</td>
    <td>48.72</td>
    <td>67.86</td>
    <td>33.28</td>
    <td>62.34</td>
    <td>73.14</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td>
    <td>52.35</td>
    <td>38.67</td>
    <td>56.18</td>
    <td>16.61</td>
    <td>69.09</td>
    <td>74.44</td>
    <td>33.06</td>
    <td>33.38</td>
    <td>53.01</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td>
    <td>62.17</td>
    <td>44.15</td>
    <td>62.13</td>
    <td>15.73</td>
    <td>22.48</td>
    <td>52.91</td>
    <td>28.23</td>
    <td>26.17</td>
    <td>45.70</td>
  </tr>
  <tr>
    <td>BLIP3</td>
    <td>55.56</td>
    <td>41.23</td>
    <td>59.49</td>
    <td>17.29</td>
    <td>50.64</td>
    <td>67.85</td>
    <td>31.22</td>
    <td>34.50</td>
    <td>50.61</td>
  </tr>
  <tr>
    <th>Gemini</th>
    <td>58.22</td>
    <td>47.47</td>
    <td>63.32</td>
    <td>16.13</td>
    <td>51.89</td>
    <td>68.52</td>
    <td>45.24</td>
    <td>28.63</td>
    <td>51.14</td>
  </tr>
  <tr>
    <th>GPT-4V</th>
    <td>52.43</td>
    <td>34.46</td>
    <td>54.67</td>
    <td>15.90</td>
    <td>51.43</td>
    <td>65.83</td>
    <td>17.22</td>
    <td>24.44</td>
    <td>54.64</td>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <td>59.85</td>
    <td>47.01</td>
    <td>62.76</td>
    <td>16.78</td>
    <td>78.54</td>
    <td>76.67</td>
    <td>25.01</td>
    <td>41.72</td>
    <td>64.27</td>
  </tr>
</table>

ever, consistency metrics (Con and S_C) do not show a similar trend. In the Question Rephrasing and the Image Restyling task, the large model performs better, while the trend is not the same in the Context Reasoning task. Hence, we observe that unlike accuracy, consistency does not always improve with increase in model size.

A.9 Analysis on Different Resolutions in Image Restyling task
In the Image Restyling task, we notice that different resolutions of original images might yield different levels of detail when styling the images. In Image Restyling figure, the Golden Gate has a higher resolution, so it shows more details after restyling. Conversely, the bottom row—the church—has a lower resolution and shows fewer details. To analyze the effect of detail levels for VLMs, we resize the images before performing the style transfer. We resize the images to low (224 × 224), mid (640 × 640), and high (1024 × 1024) resolutions. The results are shown in Figure 6 and Table 6. Compared with the original Image Styling task results (Table 2), low-resolution results drop significantly, especially in the Con and S_C metrics, indicating that detail level affects consistency the most. On the other hand, the mid and high-resolution results show consistent improvements, indicating that model consistency increases with an increase in resolution.

A.10 Analysis on Different entropy parameters
The temperature parameter in VLMs controls the level of randomness in the model’s output. Lower temperature yields more deterministic outputs, and higher temperature shows more diversity outputs. Hence, we analyze how temperature affects the consistency output in three tasks. We show the results on 3 different temperatures in Figure 7 and the actual numbers in Table 7. We set the temperatures to 0.2, 0.7, and 1 on all open-sourced models. We notice that mPLUG-Owl2 and MoE-LLaVa model performances drop significantly when the temperature increases while Qwen-VL-Chat show more consistent results.

A.11 More Analysis
We provide a deeper analysis of the Image Restyling and Context Reasoning tasks. In the Image Restyling task, we assess performance across 4 styles: Candy, Mosaic, Udnie, and Grayscale. According to Table 8, all VLMs achieve their best results with the Grayscale transformation, which is expected as this style minimally changes the original image. Conversely, the Mosaic style proves challenging for all models, likely due to its significant modification of object appearances, leading to potential confusion for the VLMs.

In the context reasoning task, we assess performance based on masking colors, lines, and shapes. Table 9 displays the results for various masking colors, with Black outperforming other colors. This could be attributed to the frequent use of black as a bounding box or mask in existing datasets, making VLMs more adept at handling black masks. Table 10 examines the effect of different numbers of masking lines, where masks with three lines perform the best, in contrast to those with only one line. This difference may arise from the area covered by the masks or the additional cues multiple lines provide about the underlying objects, aiding VLMs in making predictions. According to Table 11, open-source VLMs show a preference for Ellipse masks over Rectangles, potentially due to the smaller coverage area of ellipses, similar to the influence of the number of lines.

From these observations, it appears that mask color has a minimal impact on performance. Instead, the area covered by the mask plays a more crucial role, suggesting that the inferencing capability of VLMs could be further improved by addressing their sensitivity to the extent of occlusion.
Table 5: Impact of Model Size. Different VLM sizes on MM-R^2 benchmark. The ^ denotes that we ignore generated responses that have an empty output.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Question Rephrasing</th>
    <th colspan="3">Image Restyling</th>
    <th colspan="3">Context Reasoning</th>
  </tr>
  <tr>
    <th>Acc</th><th>S_{GT}</th><th>Con</th>
    <th>Acc</th><th>S_{GT}</th><th>Con</th>
    <th>Acc</th><th>S_{GT}</th><th>Con</th><th>S_{Con}</th>
  </tr>
  <tr>
    <td>BLIP-2 (opt2-7B)</td>
    <td>19.0*</td><td>47.3*</td><td>39.1*</td>
    <td>54.3*</td><td>11.6*</td><td>55.5*</td>
    <td>48.5*</td><td>58.0*</td><td>27.8</td><td>39.0</td><td>76.2</td><td>86.2</td>
  </tr>
  <tr>
    <td>BLIP-2 (flant5xxl)</td>
    <td>25.1</td><td>52.9</td><td>48.2</td>
    <td>63.9</td><td>13.0</td><td>17.0</td>
    <td>38.4</td><td>62.8</td><td>27.9</td><td>39.0</td><td>82.4</td><td>88.8</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M (7B)</td>
    <td>31.0</td><td>62.9</td><td>48.5</td>
    <td>64.0</td><td>10.5</td><td>15.5</td>
    <td>50.1</td><td>68.5</td><td>28.7</td><td>42.5</td><td>68.0</td><td>77.0</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M (13B)</td>
    <td>33.0</td><td>63.7</td><td>49.5</td>
    <td>64.5</td><td>10.6</td><td>16.1</td>
    <td>67.4</td><td>75.5</td><td>34.9</td><td>45.0</td><td>64.6</td><td>74.5</td>
  </tr>
</table>

![Four line plots showing performance on image restyling across different resolutions for various models](page_104_370_1247_246.png)

Figure 6: Stylization with different resolutions on the Image Restyling Task.

A.12 Consistency Improvement Details and More Results

Implementation Details. For both BLIP-2 and LLaVa 1.5M experiments, the Bi-LSTM includes 2 recurrent layers with the hidden size is set to 4096 and dropout rate 0. We follow the same setting as in (Newman et al., 2022) to set the prefix size to 3. The model is optimized by CrossEntropyLoss. The initial linear rate is set to \( 1e - 5 \). A linear decay learning rate schedule is also used. We conduct all experiments on a Nvidia A40 GPU with batch size 2 on both models. The adapter is added on the top of BLIP-2 blip2-t5 pretrain-flant5xxl and LLaVa 1.5M llava-v1.5-7b models.

Ablation Studies. We provide ablation studies in Table 12. Adapt (Sep) indicates that the adapter model is trained separately for each task, meaning each task has its own corresponding model. LoRA refers to an alternative approach where, instead of using an adapter to help VLMs handle input variability, we fine-tune the VLMs using LoRA. In the LoRA fine-tuning setting, we not only use the datasets mentioned in the LLaVA 1.5M paper but also incorporate data from the MM^3 benchmark.

A.13 Qualitative Results

Question Rephrasing We show more qualitative results of the question rephrasing task in Figure 8.

Image Restyling The qualitative results of the image restyling task are depicted in Figure 9 and Figure 10. In this task, the BLIP-2 model outperforms other VLMs, particularly in terms of consistency. Although LLaVa 1.5M and MoE-LLaVa may not always provide responses that align perfectly with the ground truth, their answers remain semantically consistent across various styles. This consistency is evident in their strong performance on metrics such as Consistency and S_C.

Context Reasoning Figure 11 and Figure 12 shows additional examples from the context reasoning task, where most VLMs generally yield similar responses as the ground truth. However, it is observed that models like Owen-VL, MoE-LLaVa, and BLIP-3 are more influenced by the presence of masks, often incorporating the mask’s color or shape into their answers. Another noteworthy trend is GPT-4V’s tendency to respond with "I cannot provide..." when the masks obscure a significant portion of the objects, indicating a threshold of visual information required for it to generate confident responses.
Table 6: Stylization with different resolutions on the Image Restyling Task.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">224 × 224</th>
    <th colspan="4">640 × 640</th>
    <th colspan="4">1024 × 1024</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>10.6</td><td>16.9</td><td>22.3</td><td>53.4</td>
    <td>15.0</td><td>17.0</td><td>42.4</td><td>65.2</td>
    <td>15.8</td><td>17.2</td><td>48.2</td><td>67.0</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>6.4</td><td>15.6</td><td>12.9</td><td>51.3</td>
    <td>9.8</td><td>15.5</td><td>47.8</td><td>60.1</td>
    <td>15.3</td><td>20.1</td><td>60.1</td><td>60.3</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>6.4</td><td>15.8</td><td>31.3</td><td>59.5</td>
    <td>12.6</td><td>15.3</td><td>55.5</td><td>71.1</td>
    <td>13.3</td><td>15.3</td><td>60.6</td><td>73.3</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>10.3</td><td>16.8</td><td>41.6</td><td>65.1</td>
    <td>14.2</td><td>16.7</td><td>69.7</td><td>74.8</td>
    <td>14.7</td><td>16.8</td><td>75.4</td><td>76.5</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>5.8</td><td>15.6</td><td>9.1</td><td>40.9</td>
    <td>9.5</td><td>15.8</td><td>20.3</td><td>52.4</td>
    <td>10.4</td><td>15.9</td><td>25.3</td><td>55.6</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>9.0</td><td>16.9</td><td>25.0</td><td>55.7</td>
    <td>13.5</td><td>17.5</td><td>53.3</td><td>69.0</td>
    <td>14.2</td><td>17.8</td><td>61.9</td><td>72.0</td>
  </tr>
</table>

Table 7: Different entropy parameters on three tasks.

Question Rephrasing Task

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">0.2</th>
    <th colspan="4">0.7</th>
    <th colspan="4">1</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>25.1</td><td>52.9</td><td>48.2</td><td>63.9</td>
    <td>25.1</td><td>52.9</td><td>48.2</td><td>63.9</td>
    <td>25.1</td><td>52.9</td><td>48.2</td><td>63.9</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>33.2</td><td>63.8</td><td>48.6</td><td>63.7</td>
    <td>30.4</td><td>61.1</td><td>36.7</td><td>55.6</td>
    <td>27.7</td><td>58.1</td><td>30.7</td><td>51.3</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>31.2</td><td>62.6</td><td>46.0</td><td>62.4</td>
    <td>26.9</td><td>59.2</td><td>32.5</td><td>53.8</td>
    <td>10.3</td><td>19.3</td><td>34.3</td><td>62.3</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>28.8</td><td>58.0</td><td>45.3</td><td>61.2</td>
    <td>26.3</td><td>56.9</td><td>34.8</td><td>54.7</td>
    <td>24.2</td><td>54.5</td><td>26.4</td><td>49.4</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>36.7</td><td>70.6</td><td>56.4</td><td>70.4</td>
    <td>36.3</td><td>70.2</td><td>55.3</td><td>69.7</td>
    <td>36.3</td><td>70.2</td><td>55.3</td><td>69.7</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>30.9</td><td>61.0</td><td>47.7</td><td>64.0</td>
    <td>30.9</td><td>61.0</td><td>47.7</td><td>64.0</td>
    <td>30.9</td><td>61.0</td><td>47.7</td><td>64.0</td>
  </tr>
</table>

Image Restyling Task

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">0.2</th>
    <th colspan="4">0.7</th>
    <th colspan="4">1</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>13.0</td><td>17.0</td><td>38.4</td><td>62.8</td>
    <td>13.0</td><td>17.0</td><td>38.4</td><td>62.8</td>
    <td>13.0</td><td>17.0</td><td>38.4</td><td>62.8</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>9.6</td><td>15.4</td><td>46.1</td><td>66.6</td>
    <td>9.0</td><td>15.2</td><td>26.3</td><td>59.2</td>
    <td>8.4</td><td>14.9</td><td>13.5</td><td>53.8</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>10.3</td><td>15.5</td><td>44.4</td><td>60.0</td>
    <td>9.6</td><td>14.9</td><td>19.0</td><td>56.9</td>
    <td>10.5</td><td>15.5</td><td>50.1</td><td>68.5</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>12.9</td><td>16.6</td><td>70.3</td><td>74.8</td>
    <td>11.9</td><td>16.2</td><td>48.0</td><td>67.5</td>
    <td>11.7</td><td>15.7</td><td>28.2</td><td>62.0</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>8.4</td><td>15.8</td><td>24.4</td><td>54.1</td>
    <td>8.2</td><td>15.7</td><td>23.1</td><td>53.5</td>
    <td>8.2</td><td>15.7</td><td>23.1</td><td>53.3</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>11.9</td><td>17.5</td><td>51.4</td><td>68.2</td>
    <td>11.9</td><td>17.5</td><td>51.4</td><td>68.2</td>
    <td>11.9</td><td>17.5</td><td>51.4</td><td>68.2</td>
  </tr>
</table>

Context Reasoning Task

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">0.2</th>
    <th colspan="4">0.7</th>
    <th colspan="4">1</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>27.9</td><td>39.0</td><td>82.5</td><td>88.8</td>
    <td>27.9</td><td>39.0</td><td>82.5</td><td>88.8</td>
    <td>27.9</td><td>39.0</td><td>82.4</td><td>88.8</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>27.4</td><td>41.0</td><td>55.4</td><td>67.0</td>
    <td>24.5</td><td>35.3</td><td>27.6</td><td>48.0</td>
    <td>22.1</td><td>30.1</td><td>10.5</td><td>37.2</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>28.2</td><td>42.0</td><td>62.4</td><td>72.4</td>
    <td>20.1</td><td>28.9</td><td>25.9</td><td>42.3</td>
    <td>22.9</td><td>37.8</td><td>15.4</td><td>41.6</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>25.2</td><td>38.5</td><td>39.4</td><td>56.9</td>
    <td>24.4</td><td>34.7</td><td>22.0</td><td>45.1</td>
    <td>23.8</td><td>31.6</td><td>10.0</td><td>38.7</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>20.4</td><td>32.2</td><td>32.9</td><td>50.6</td>
    <td>20.1</td><td>32.2</td><td>30.7</td><td>49.0</td>
    <td>20.1</td><td>32.2</td><td>30.7</td><td>49.0</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>28.0</td><td>36.5</td><td>40.0</td><td>54.9</td>
    <td>28.0</td><td>36.5</td><td>40.0</td><td>54.9</td>
    <td>28.0</td><td>36.5</td><td>40.0</td><td>54.9</td>
  </tr>
</table>

![Impact of Entropy. Different entropy parameters on all three tasks for models tested.](page_1012_1342_1017_180.png)

Figure 7: Impact of Entropy. Different entropy parameters on all three tasks for models tested.
Table 8: Results across different styles in the Image Restyling Task.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">Candy</th>
    <th colspan="6">Mosaic</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>Gt</sub></th>
    <th>Con</th>
    <th>Sc</th>
    <th>Acc</th>
    <th>S<sub>Gt</sub></th>
    <th>Con</th>
    <th>Sc</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>13.59</td><td>16.73</td><td>40.99</td><td>63.80</td><td>10.51</td><td>16.24</td><td>48.35</td><td>67.91</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>8.33</td><td>15.79</td><td>24.17</td><td>58.40</td><td>4.28</td><td>14.81</td><td>29.65</td><td>60.86</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>10.21</td><td>15.76</td><td>52.55</td><td>69.04</td><td>7.51</td><td>15.53</td><td>55.78</td><td>70.79</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>13.59</td><td>17.21</td><td>69.74</td><td>74.45</td><td>9.08</td><td>17.00</td><td>74.70</td><td>76.47</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>7.51</td><td>15.88</td><td>20.27</td><td>51.69</td><td>3.45</td><td>15.00</td><td>29.35</td><td>57.96</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>12.84</td><td>17.66</td><td>56.31</td><td>70.15</td><td>7.81</td><td>17.60</td><td>58.41</td><td>71.48</td>
  </tr>
  <tr>
    <td>Gemini</td><td>14.11</td><td>16.58</td><td>51.88</td><td>68.63</td><td>10.21</td><td>16.50</td><td>59.01</td><td>70.75</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>11.11</td><td>16.91</td><td>57.64</td><td>67.58</td><td>7.64</td><td>15.70</td><td>59.03</td><td>69.05</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>15.84</td><td>17.55</td><td>81.23</td><td>77.61</td><td>12.69</td><td>17.15</td><td>86.56</td><td>79.31</td>
  </tr>
  <tr>
    <th colspan="10">Failure Cases</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>-</td><td>16.59</td><td>39.90</td><td>63.04</td><td>-</td><td>16.11</td><td>46.55</td><td>66.98</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>-</td><td>15.76</td><td>23.50</td><td>58.09</td><td>-</td><td>14.78</td><td>28.73</td><td>60.55</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>-</td><td>15.71</td><td>51.27</td><td>68.48</td><td>-</td><td>15.48</td><td>54.05</td><td>70.11</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>-</td><td>17.23</td><td>68.74</td><td>74.03</td><td>-</td><td>16.97</td><td>73.51</td><td>76.13</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>-</td><td>15.91</td><td>19.59</td><td>51.28</td><td>-</td><td>15.00</td><td>28.80</td><td>57.61</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>-</td><td>17.69</td><td>55.22</td><td>69.81</td><td>-</td><td>17.57</td><td>58.62</td><td>71.30</td>
  </tr>
  <tr>
    <td>Gemini</td><td>-</td><td>16.58</td><td>56.16</td><td>68.20</td><td>-</td><td>16.43</td><td>57.40</td><td>70.25</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>-</td><td>16.96</td><td>56.43</td><td>67.18</td><td>-</td><td>15.68</td><td>57.86</td><td>68.57</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>-</td><td>17.51</td><td>79.52</td><td>77.05</td><td>-</td><td>17.02</td><td>85.63</td><td>78.93</td>
  </tr>
  <tr>
    <th colspan="10">Udnie</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>Gt</sub></th>
    <th>Con</th>
    <th>Sc</th>
    <th>Acc</th>
    <th>S<sub>Gt</sub></th>
    <th>Con</th>
    <th>Sc</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>10.96</td><td>16.78</td><td>36.94</td><td>62.32</td><td>16.07</td><td>18.32</td><td>40.69</td><td>63.71</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>7.96</td><td>14.90</td><td>29.43</td><td>60.53</td><td>15.24</td><td>15.47</td><td>21.92</td><td>57.31</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>9.91</td><td>15.36</td><td>61.17</td><td>67.11</td><td>15.17</td><td>15.32</td><td>51.20</td><td>69.87</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>12.31</td><td>16.24</td><td>68.99</td><td>74.43</td><td>16.74</td><td>15.95</td><td>70.42</td><td>74.31</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>7.43</td><td>15.45</td><td>22.37</td><td>54.13</td><td>14.71</td><td>16.57</td><td>20.95</td><td>50.74</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>10.29</td><td>16.50</td><td>47.52</td><td>67.19</td><td>16.74</td><td>17.47</td><td>52.63</td><td>68.06</td>
  </tr>
  <tr>
    <td>Gemini</td><td>13.21</td><td>15.98</td><td>55.56</td><td>69.66</td><td>17.19</td><td>15.54</td><td>50.83</td><td>68.70</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>6.25</td><td>15.59</td><td>47.22</td><td>66.38</td><td>13.89</td><td>15.38</td><td>52.08</td><td>66.11</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>16.59</td><td>16.53</td><td>77.40</td><td>76.09</td><td>20.12</td><td>16.09</td><td>79.80</td><td>77.22</td>
  </tr>
  <tr>
    <th colspan="10">Grayscale</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>Gt</sub></th>
    <th>Con</th>
    <th>Sc</th>
    <th>Acc</th>
    <th>S<sub>Gt</sub></th>
    <th>Con</th>
    <th>Sc</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>-</td><td>16.64</td><td>35.77</td><td>61.53</td><td>-</td><td>18.15</td><td>39.42</td><td>62.84</td>
  </tr>
  <tr>
    <td>mPLUG-Owl2</td><td>-</td><td>14.97</td><td>28.65</td><td>60.22</td><td>-</td><td>15.40</td><td>20.89</td><td>56.96</td>
  </tr>
  <tr>
    <td>LLaVa 1.5M</td><td>-</td><td>15.29</td><td>44.60</td><td>66.43</td><td>-</td><td>15.24</td><td>49.84</td><td>69.31</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>-</td><td>16.27</td><td>67.61</td><td>74.06</td><td>-</td><td>15.99</td><td>69.47</td><td>74.06</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>-</td><td>15.43</td><td>21.66</td><td>53.79</td><td>-</td><td>16.58</td><td>20.43</td><td>50.35</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>-</td><td>16.49</td><td>46.47</td><td>66.86</td><td>-</td><td>17.43</td><td>51.95</td><td>67.75</td>
  </tr>
  <tr>
    <td>Gemini</td><td>-</td><td>15.95</td><td>53.70</td><td>69.19</td><td>-</td><td>15.57</td><td>49.68</td><td>68.36</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>-</td><td>15.38</td><td>43.71</td><td>66.06</td><td>-</td><td>15.36</td><td>50.71</td><td>65.39</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>-</td><td>16.50</td><td>75.64</td><td>78.57</td><td>-</td><td>16.10</td><td>78.78</td><td>76.94</td>
  </tr>
</table>
Table 9: Results across different masking color in the Context Reasoning Task.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">Blue</th>
    <th colspan="6">Red</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>G</sub>T</th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>Acc</th>
    <th>S<sub>G</sub>T</th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td rowspan="10">Failure Cases</td>
    <td>BLIP-2</td><td>26.67</td><td>38.27</td><td>82.70</td><td>89.50</td><td>27.88</td><td>39.09</td><td>79.89</td><td>87.71</td>
  </tr>
  <tr><td>mPLUG-Owl2</td><td>23.51</td><td>34.62</td><td>26.07</td><td>46.06</td><td>23.06</td><td>35.26</td><td>27.88</td><td>48.38</td></tr>
  <tr><td>LLaVa 1.5M</td><td>26.31</td><td>41.51</td><td>69.18</td><td>77.44</td><td>27.88</td><td>42.22</td><td>65.82</td><td>75.70</td></tr>
  <tr><td>MoE-LLaVa</td><td>20.35</td><td>35.10</td><td>37.37</td><td>56.10</td><td>23.06</td><td>37.37</td><td>34.81</td><td>54.81</td></tr>
  <tr><td>Owen-VL-Chat</td><td>19.85</td><td>31.78</td><td>35.50</td><td>47.14</td><td>20.99</td><td>29.97</td><td>29.05</td><td>47.80</td></tr>
  <tr><td>BLIP-3</td><td>23.75</td><td>35.44</td><td>39.71</td><td>54.30</td><td>28.02</td><td>37.93</td><td>40.08</td><td>54.84</td></tr>
  <tr><td>Gemini</td><td>54.95</td><td>56.52</td><td>44.81</td><td>62.90</td><td>51.54</td><td>54.42</td><td>43.89</td><td>61.95</td></tr>
  <tr><td>GPT-4V</td><td>32.22</td><td>20.39</td><td>30.56</td><td>56.53</td><td>28.83</td><td>18.83</td><td>33.74</td><td>56.90</td></tr>
  <tr><td>GPT-4o</td><td>50.06</td><td>31.26</td><td>48.11</td><td>66.51</td><td>52.41</td><td>31.80</td><td>47.05</td><td>66.06</td></tr>
  <tr><td>All</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr>
    <td rowspan="10">Failure Cases</td>
    <td>BLIP-2</td><td>27.56</td><td>78.04</td><td>87.26</td><td>-</td><td>29.21</td><td>74.79</td><td>85.05</td><td>-</td>
  </tr>
  <tr><td>mPLUG-Owl2</td><td>30.85</td><td>21.21</td><td>42.46</td><td>-</td><td>30.05</td><td>24.12</td><td>45.39</td><td>-</td></tr>
  <tr><td>LLaVa 1.5M</td><td>32.49</td><td>41.05</td><td>53.73</td><td>-</td><td>33.10</td><td>59.93</td><td>54.72</td><td>-</td></tr>
  <tr><td>MoE-LLaVa</td><td>30.65</td><td>31.63</td><td>52.76</td><td>-</td><td>32.01</td><td>30.04</td><td>51.16</td><td>-</td></tr>
  <tr><td>Owen-VL-Chat</td><td>27.21</td><td>28.76</td><td>46.66</td><td>-</td><td>26.11</td><td>24.17</td><td>44.60</td><td>-</td></tr>
  <tr><td>BLIP-3</td><td>30.47</td><td>35.61</td><td>50.71</td><td>-</td><td>32.45</td><td>34.56</td><td>50.19</td><td>-</td></tr>
  <tr><td>Gemini</td><td>45.78</td><td>28.33</td><td>51.84</td><td>-</td><td>44.08</td><td>30.48</td><td>52.99</td><td>-</td></tr>
  <tr><td>GPT-4V</td><td>17.09</td><td>26.11</td><td>54.20</td><td>-</td><td>13.54</td><td>28.26</td><td>54.14</td><td>-</td></tr>
  <tr><td>GPT-4o</td><td>25.15</td><td>41.40</td><td>64.35</td><td>-</td><td>24.66</td><td>39.52</td><td>63.38</td><td>-</td></tr>
  <tr><td>All</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr>
    <th colspan="6">Green</th>
    <th colspan="6">Yellow</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>G</sub>T</th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>Acc</th>
    <th>S<sub>G</sub>T</th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td rowspan="10">Failure Cases</td>
    <td>BLIP-2</td><td>27.76</td><td>39.54</td><td>85.44</td><td>90.36</td><td>28.23</td><td>38.64</td><td>83.60</td><td>89.46</td>
  </tr>
  <tr><td>mPLUG-Owl2</td><td>23.13</td><td>34.06</td><td>24.90</td><td>45.90</td><td>25.00</td><td>35.72</td><td>28.23</td><td>48.94</td></tr>
  <tr><td>LLaVa 1.5M</td><td>28.30</td><td>42.14</td><td>67.48</td><td>76.50</td><td>26.88</td><td>41.57</td><td>70.03</td><td>78.57</td></tr>
  <tr><td>MoE-LLaVa</td><td>22.99</td><td>36.26</td><td>39.46</td><td>56.35</td><td>19.89</td><td>34.29</td><td>39.65</td><td>57.32</td></tr>
  <tr><td>Owen-VL-Chat</td><td>18.78</td><td>31.92</td><td>30.20</td><td>48.70</td><td>18.66</td><td>30.60</td><td>32.12</td><td>50.19</td></tr>
  <tr><td>BLIP-3</td><td>27.31</td><td>36.12</td><td>52.90</td><td>55.22</td><td>21.19</td><td>32.42</td><td>34.32</td><td>54.32</td></tr>
  <tr><td>Gemini</td><td>57.96</td><td>57.28</td><td>45.71</td><td>52.09</td><td>52.76</td><td>55.19</td><td>45.76</td><td>63.16</td></tr>
  <tr><td>GPT-4V</td><td>30.49</td><td>21.32</td><td>37.20</td><td>57.86</td><td>33.11</td><td>21.01</td><td>38.41</td><td>59.81</td></tr>
  <tr><td>GPT-4o</td><td>50.88</td><td>31.49</td><td>51.70</td><td>67.16</td><td>51.21</td><td>31.64</td><td>50.54</td><td>67.32</td></tr>
  <tr><td>All</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr>
    <th colspan="6">White</th>
    <th colspan="6">Black</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Acc</th>
    <th>S<sub>G</sub>T</th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
    <th>Acc</th>
    <th>S<sub>G</sub>T</th>
    <th>Con</th>
    <th>S<sub>C</sub></th>
  </tr>
  <tr>
    <td rowspan="10">Failure Cases</td>
    <td>BLIP-2</td><td>27.40</td><td>38.14</td><td>81.92</td><td>88.02</td><td>29.59</td><td>40.14</td><td>81.11</td><td>87.64</td>
  </tr>
  <tr><td>mPLUG-Owl2</td><td>24.68</td><td>34.62</td><td>30.13</td><td>49.65</td><td>27.48</td><td>37.71</td><td>28.93</td><td>49.13</td></tr>
  <tr><td>LLaVa 1.5M</td><td>27.98</td><td>42.03</td><td>68.01</td><td>76.96</td><td>34.74</td><td>45.64</td><td>67.64</td><td>76.87</td></tr>
  <tr><td>MoE-LLaVa</td><td>30.42</td><td>41.84</td><td>41.61</td><td>58.18</td><td>34.87</td><td>46.00</td><td>42.93</td><td>58.54</td></tr>
  <tr><td>Owen-VL-Chat</td><td>21.66</td><td>33.11</td><td>31.85</td><td>49.50</td><td>24.04</td><td>35.80</td><td>27.34</td><td>47.30</td></tr>
  <tr><td>BLIP-3</td><td>31.13</td><td>37.06</td><td>41.46</td><td>56.17</td><td>37.38</td><td>38.21</td><td>38.57</td><td>53.86</td></tr>
  <tr><td>Gemini</td><td>57.10</td><td>56.42</td><td>46.62</td><td>62.11</td><td>57.09</td><td>46.44</td><td>63.04</td><td>65.30</td></tr>
  <tr><td>GPT-4V</td><td>30.15</td><td>21.81</td><td>38.58</td><td>57.89</td><td>25.05</td><td>22.86</td><td>31.37</td><td>55.30</td></tr>
  <tr><td>GPT-4o</td><td>53.66</td><td>32.04</td><td>50.22</td><td>67.35</td><td>52.44</td><td>31.84</td><td>49.54</td><td>66.58</td></tr>
  <tr><td>All</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr>
    <td rowspan="10">Failure Cases</td>
    <td>BLIP-2</td><td>-</td><td>28.85</td><td>77.26</td><td>85.57</td><td>-</td><td>29.33</td><td>76.30</td><td>85.14</td>
  </tr>
  <tr><td>mPLUG-Owl2</td><td>-</td><td>30.26</td><td>25.60</td><td>46.48</td><td>-</td><td>33.81</td><td>25.18</td><td>46.32</td></tr>
  <tr><td>LLaVa 1.5M</td><td>-</td><td>32.58</td><td>62.14</td><td>73.10</td><td>-</td><td>37.15</td><td>61.33</td><td>72.64</td></tr>
  <tr><td>MoE-LLaVa</td><td>-</td><td>36.67</td><td>35.17</td><td>53.95</td><td>-</td><td>40.16</td><td>36.20</td><td>53.80</td></tr>
  <tr><td>Owen-VL-Chat</td><td>-</td><td>29.05</td><td>29.39</td><td>46.17</td><td>-</td><td>32.35</td><td>32.34</td><td>44.20</td></tr>
  <tr><td>BLIP-3</td><td>-</td><td>32.44</td><td>38.66</td><td>51.84</td><td>-</td><td>31.55</td><td>48.88</td><td>51.88</td></tr>
  <tr><td>Gemini</td><td>-</td><td>46.09</td><td>28.35</td><td>50.30</td><td>-</td><td>46.44</td><td>28.63</td><td>50.75</td></tr>
  <tr><td>GPT-4V</td><td>-</td><td>18.44</td><td>32.58</td><td>55.81</td><td>-</td><td>18.44</td><td>25.78</td><td>53.19</td></tr>
  <tr><td>GPT-4o</td><td>-</td><td>26.26</td><td>40.13</td><td>64.92</td><td>-</td><td>24.94</td><td>42.16</td><td>63.82</td></tr>
  <tr><td>All</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
</table>
Table 10: Results across different numbers of lines in the Context Reasoning Task.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">1</th>
    <th colspan="3">3</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>Sc</th><th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>Sc</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>23.71</td><td>36.08</td><td>78.44</td><td>86.34</td><td>32.10</td><td>41.27</td><td>82.75</td><td>88.91</td>
  </tr>
  <tr>
    <td>mPLUG-Owl12</td><td>19.50</td><td>31.66</td><td>25.09</td><td>45.49</td><td>29.43</td><td>38.16</td><td>30.17</td><td>48.93</td>
  </tr>
  <tr>
    <td>LLaVA 1.5M</td><td>23.88</td><td>38.60</td><td>61.86</td><td>72.55</td><td>33.76</td><td>46.20</td><td>69.83</td><td>78.16</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>17.27</td><td>32.56</td><td>32.99</td><td>53.31</td><td>30.54</td><td>42.35</td><td>42.99</td><td>58.78</td>
  </tr>
  <tr>
    <td>All</td><td>13.06</td><td>25.69</td><td>26.37</td><td>45.45</td><td>28.78</td><td>37.72</td><td>33.95</td><td>50.83</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>15.81</td><td>30.66</td><td>34.19</td><td>50.24</td><td>37.82</td><td>41.47</td><td>43.82</td><td>58.42</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>55.98</td><td>56.17</td><td>42.73</td><td>61.62</td><td>54.24</td><td>56.60</td><td>47.51</td><td>63.40</td>
  </tr>
  <tr>
    <td>Gemini</td><td>55.25</td><td>20.33</td><td>32.38</td><td>55.62</td><td>33.31</td><td>21.66</td><td>34.17</td><td>58.25</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>46.65</td><td>29.32</td><td>48.88</td><td>66.38</td><td>56.83</td><td>33.65</td><td>49.91</td><td>66.88</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>BLIP-2</td><td>-</td><td>26.47</td><td>73.36</td><td>83.62</td><td>-</td><td>30.24</td><td>77.53</td><td>86.15</td>
  </tr>
  <tr>
    <td>mPLUG-Owl12</td><td>-</td><td>27.70</td><td>20.96</td><td>42.48</td><td>-</td><td>33.19</td><td>25.05</td><td>45.25</td>
  </tr>
  <tr>
    <td>LLaVA 1.5M</td><td>-</td><td>29.90</td><td>55.95</td><td>68.54</td><td>-</td><td>36.36</td><td>63.39</td><td>73.96</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>-</td><td>27.70</td><td>27.32</td><td>49.79</td><td>-</td><td>36.17</td><td>36.06</td><td>54.40</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>-</td><td>22.18</td><td>22.15</td><td>42.44</td><td>-</td><td>32.94</td><td>28.18</td><td>46.68</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>-</td><td>26.00</td><td>24.45</td><td>46.74</td><td>-</td><td>35.35</td><td>37.83</td><td>53.51</td>
  </tr>
  <tr>
    <td>Gemini</td><td>-</td><td>45.94</td><td>45.78</td><td>52.20</td><td>-</td><td>48.38</td><td>51.12</td><td>59.80</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>-</td><td>15.74</td><td>27.67</td><td>52.91</td><td>-</td><td>18.82</td><td>30.70</td><td>56.64</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>-</td><td>22.48</td><td>40.88</td><td>63.66</td><td>-</td><td>26.96</td><td>40.42</td><td>64.06</td>
  </tr>
  <tr>
    <td>Failure Cases</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>BLIP-2</td><td>29.90</td><td>40.23</td><td>84.28</td><td>89.98</td><td>26.29</td><td>38.51</td><td>84.45</td><td>90.05</td>
  </tr>
  <tr>
    <td>mPLUG-Owl12</td><td>26.02</td><td>36.44</td><td>59.92</td><td>48.63</td><td>23.32</td><td>35.34</td><td>28.56</td><td>48.92</td>
  </tr>
  <tr>
    <td>LLaVA 1.5M</td><td>30.08</td><td>43.75</td><td>70.82</td><td>78.99</td><td>27.34</td><td>41.81</td><td>69.96</td><td>78.56</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>28.09</td><td>40.00</td><td>39.93</td><td>57.33</td><td>25.24</td><td>39.32</td><td>42.01</td><td>58.20</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>19.87</td><td>33.27</td><td>32.25</td><td>50.75</td><td>19.30</td><td>32.57</td><td>30.48</td><td>48.98</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>31.53</td><td>37.85</td><td>40.38</td><td>54.29</td><td>27.51</td><td>36.35</td><td>42.01</td><td>56.80</td>
  </tr>
  <tr>
    <td>Gemini</td><td>58.42</td><td>57.86</td><td>46.11</td><td>63.49</td><td>52.23</td><td>54.45</td><td>44.53</td><td>61.93</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>31.19</td><td>21.95</td><td>33.32</td><td>57.87</td><td>27.52</td><td>20.22</td><td>36.82</td><td>58.01</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>54.56</td><td>33.07</td><td>49.32</td><td>67.00</td><td>49.34</td><td>30.81</td><td>49.87</td><td>67.03</td>
  </tr>
  <tr>
    <td>Failure Cases</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>BLIP-2</td><td>-</td><td>30.10</td><td>79.95</td><td>87.98</td><td>-</td><td>28.21</td><td>80.52</td><td>88.05</td>
  </tr>
  <tr>
    <td>mPLUG-Owl12</td><td>-</td><td>32.55</td><td>23.18</td><td>45.74</td><td>-</td><td>31.04</td><td>24.13</td><td>45.87</td>
  </tr>
  <tr>
    <td>LLaVA 1.5M</td><td>-</td><td>34.23</td><td>65.65</td><td>75.23</td><td>-</td><td>33.06</td><td>64.82</td><td>75.15</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>-</td><td>34.53</td><td>34.01</td><td>53.50</td><td>-</td><td>34.27</td><td>36.55</td><td>54.58</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>-</td><td>29.47</td><td>28.03</td><td>47.85</td><td>-</td><td>28.91</td><td>26.67</td><td>46.08</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>-</td><td>32.71</td><td>34.37</td><td>49.77</td><td>-</td><td>31.24</td><td>36.89</td><td>52.85</td>
  </tr>
  <tr>
    <td>Gemini</td><td>-</td><td>47.07</td><td>30.64</td><td>52.25</td><td>-</td><td>43.07</td><td>29.02</td><td>51.01</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>-</td><td>17.66</td><td>30.16</td><td>53.79</td><td>-</td><td>16.06</td><td>32.11</td><td>55.54</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>-</td><td>26.53</td><td>43.16</td><td>64.69</td><td>-</td><td>24.40</td><td>42.44</td><td>64.70</td>
  </tr>
  <tr>
    <td>Failure Cases</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 11: Results across different masking shapes in the Context Reasoning Task.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Rectangle</th>
    <th colspan="3">Ellipse</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>Sc</th><th>Acc</th><th>S<sub>GT</sub></th><th>Con</th><th>Sc</th>
  </tr>
  <tr>
    <td>BLIP-2</td><td>26.51</td><td>38.32</td><td>82.66</td><td>88.98</td><td>29.22</td><td>39.58</td><td>82.24</td><td>88.63</td>
  </tr>
  <tr>
    <td>mPLUG-Owl12</td><td>22.34</td><td>34.12</td><td>27.39</td><td>48.01</td><td>26.47</td><td>36.47</td><td>27.89</td><td>47.93</td>
  </tr>
  <tr>
    <td>LLaVA 1.5M</td><td>26.42</td><td>41.06</td><td>67.84</td><td>72.54</td><td>30.78</td><td>46.78</td><td>68.23</td><td>76.80</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>20.18</td><td>35.60</td><td>57.99</td><td>57.01</td><td>29.74</td><td>31.17</td><td>56.38</td><td>56.81</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>16.33</td><td>29.24</td><td>27.89</td><td>47.01</td><td>23.66</td><td>34.99</td><td>33.32</td><td>50.77</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>24.13</td><td>35.57</td><td>40.83</td><td>55.29</td><td>31.55</td><td>37.34</td><td>39.27</td><td>54.48</td>
  </tr>
  <tr>
    <td>Gemini</td><td>54.39</td><td>56.20</td><td>45.06</td><td>62.34</td><td>55.98</td><td>56.30</td><td>45.27</td><td>62.8</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>33.19</td><td>21.16</td><td>34.48</td><td>57.16</td><td>30.43</td><td>20.83</td><td>34.89</td><td>57.69</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>47.71</td><td>30.18</td><td>50.46</td><td>67.07</td><td>55.52</td><td>33.07</td><td>48.58</td><td>66.58</td>
  </tr>
  <tr>
    <td>Failure Cases</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>BLIP-2</td><td>-</td><td>27.96</td><td>78.13</td><td>86.64</td><td>-</td><td>29.36</td><td>77.45</td><td>86.19</td>
  </tr>
  <tr>
    <td>mPLUG-Owl12</td><td>-</td><td>29.94</td><td>23.06</td><td>45.00</td><td>-</td><td>32.10</td><td>23.50</td><td>44.63</td>
  </tr>
  <tr>
    <td>LLaVA 1.5M</td><td>-</td><td>32.47</td><td>62.37</td><td>73.58</td><td>-</td><td>34.05</td><td>62.31</td><td>72.71</td>
  </tr>
  <tr>
    <td>MoE-LLaVa</td><td>-</td><td>30.28</td><td>33.86</td><td>53.35</td><td>-</td><td>35.71</td><td>32.93</td><td>52.68</td>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>-</td><td>26.04</td><td>24.26</td><td>44.36</td><td>-</td><td>30.36</td><td>28.02</td><td>47.01</td>
  </tr>
  <tr>
    <td>BLIP-3</td><td>-</td><td>30.24</td><td>35.64</td><td>51.27</td><td>-</td><td>32.16</td><td>33.42</td><td>49.99</td>
  </tr>
  <tr>
    <td>Gemini</td><td>-</td><td>45.49</td><td>28.64</td><td>51.24</td><td>-</td><td>44.72</td><td>28.54</td><td>50.87</td>
  </tr>
  <tr>
    <td>GPT-4V</td><td>-</td><td>16.56</td><td>29.37</td><td>54.58</td><td>-</td><td>17.53</td><td>30.95</td><td>55.82</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>-</td><td>23.60</td><td>42.97</td><td>64.80</td><td>-</td><td>26.39</td><td>40.48</td><td>63.75</td>
  </tr>
  <tr>
    <td>Failure Cases</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>
Table 12: Ablation Studies on Adapter and LoRA. Adapt (Sep) indicates that the adapter model is trained separately for each task. We also provide the LoRA fine-tuning on LLaVA 1.5M.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Question Rephrasing</th>
    <th colspan="4">Image Restyling</th>
    <th colspan="4">Context Reasoning</th>
  </tr>
  <tr>
    <th>Acc</th><th>S<sub>Gt</sub></th><th>Con</th><th>Sc</th>
    <th>Acc</th><th>S<sub>Gt</sub></th><th>Con</th><th>Sc</th>
    <th>Acc</th><th>S<sub>Gt</sub></th><th>Con</th><th>Sc</th>
  </tr>
  <tr>
    <td>Ori.</td>
    <td>BLIP-2</td>
    <td>25.1</td><td>52.9</td><td>48.2</td><td>63.9</td>
    <td>13.0</td><td>17.0</td><td>38.4</td><td>62.8</td>
    <td>27.9</td><td>39.0</td><td>82.4</td><td>88.8</td>
  </tr>
  <tr>
    <td></td>
    <td>LLaVa 1.5M</td>
    <td>26.9</td><td>59.2</td><td>32.5</td><td>53.8</td>
    <td>9.6</td><td>14.9</td><td>19.0</td><td>56.9</td>
    <td>20.1</td><td>28.9</td><td>25.9</td><td>42.3</td>
  </tr>
  <tr>
    <td>Adapt (Sep)</td>
    <td>BLIP-2</td>
    <td>27.6</td><td>64.7</td><td>61.8</td><td>73.7</td>
    <td>36.7</td><td>25.6</td><td>49.0</td><td>64.1</td>
    <td>54.6</td><td>52.4</td><td>90.1</td><td>94.7</td>
  </tr>
  <tr>
    <td></td>
    <td>LLaVa 1.5M</td>
    <td>31.4</td><td>65.9</td><td>43.2</td><td>62.3</td>
    <td>18.1</td><td>28.1</td><td>32.6</td><td>52.6</td>
    <td>58.6</td><td>73.8</td><td>62.0</td><td>75.5</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>LLaVa 1.5M</td>
    <td>28.0</td><td>62.9</td><td>40.9</td><td>59.8</td>
    <td>27.4</td><td>26.1</td><td>29.9</td><td>51.0</td>
    <td>50.5</td><td>68.8</td><td>56.3</td><td>71.6</td>
  </tr>
</table>

Prompt: Please answer the question with a very-short answer. + <R1/R2/R3>
R1) What is the designated name for the location where these benches are found?
R2) In which specific park can these benches be found?
R3) What is the official name of the park where these benches reside?

Q) What is the name of the park these benches reside in?
A) central park

![Qualitative results for the Question Rephrasing Task](page_1012_1042_1022_370.png)

Figure 8: Qualitative Results for the Question Rephrasing Task.
Prompt: Please describe the place in the image in two sentences.
GT: inside bus

BLIP-2
S1) the inside of a bus Acc: 100
S2) the inside of a bus with stained glass windows Srr: 81.7
S3) the interior of a bus Con: 50
S4) the inside of a bus Sc: 82.2

LLaVa 1.5M (Improved)
S1) bus Acc: 50
S2) inside of a bus Srr: 88.3
S3) bus Con: 100
S4) inside of a bus Sc: 85.7

BLIP-2 (Improved)
S1) inside bus Acc: 100
S2) inside bus Srr: 100
S3) inside bus Con: 100
S4) inside bus Sc: 100

S1) Candy
S2) Mosaic
S3) Udnie
S4) Grayscale

mPlug_0w2z
S1) The image shows a bus with a colorful interior, featuring various seats and a mirror on the wall.
S2) The image shows a colorful glass mosaic depicting a stained-glass train car with a colorful design, likely showcasing the train seats and windows.
S3) The image is a drawing of a passenger bus with various seating arrangements, including benches and chairs.
S4) The image shows a bus with empty seats and a blank space in the center of the front area.

moellava
S1) The image shows the inside of a bus, with a row of seats and a window. The seats are colorful and arranged in rows.
S2) The image shows the interior of a bus, with a colorful and artistic design. The seats are arranged in rows, and the overall atmosphere is vibrant and visually appealing.
S3) The image shows the interior of a bus with a row of seats, some of which are empty. The seats are covered with a blue and orange striped fabric.
S4) The image shows the interior of a bus, with rows of seats and a window. The bus is empty, and the seats are arranged in a neat and orderly fashion.

Queen-Vi
S1) A bus with a yellow and blue interior and red seats.
S2) A stained glass window showing a car with seats for five people.
S3) The inside of a bus with blue and grey seats.
S4) An empty bus with many seats and windows.

BLIP
S1) The image is a painting of the inside of a bus. The seats are red and blue and there are two speakers on the ceiling.
S2) The image depicts the interior of a bus, with rows of seats and windows. The seats are arranged in a grid-like pattern and the windows are rectangular, allowing light to filter into the bus.
S3) The image shows the inside of a bus with rows of seats.
S4) The image is a black and white photo of the inside of a bus. The bus is empty and has rows of seats on both sides of a central aisle.

Gemini
S1) This is the inside of a school bus, likely a smaller one than what is typically used for school children. The seats are patterned and lined up in rows, with some seats appearing empty and others having personal belongings on them.
S2) The image shows the interior of a bus, with the seats on either side of the aisle visible. The seats are patterned with orange and blue lines, and the windows are covered with a stained glass effect.
S3) The image shows the inside of a bus with rows of seats facing each other. The seats are covered in a colorful patterned fabric.
S4) The image shows the inside of a bus with rows of seats facing each other. There are windows on either side of the bus and the ceiling is white with a vent in the middle.

GPT-4v
S1) The image depicts the interior of a brightly colored bus, featuring a unique design with patterned seats and warm colors. Large windows line the sides, allowing natural light into the space, creating an inviting atmosphere.
S2) The image presents an abstract representation of the interior of a bus, characterized by vibrant, colorful patterns on the seats. Large windows line the sides, allowing light to wash over the intricately drawn design, creating a whimsical atmosphere.
S3) The image depicts the interior of a bus or van, featuring several rows of seats arranged in a linear fashion. The seating is designed with patterned upholstery and there are large windows visible along the sides, allowing for natural light to enter the space.
S4) The image depicts the interior of a bus or shuttle, characterized by a clean and organized seating arrangement. The seats are upholstered in a patterned fabric, and there are multiple windows along the sides allowing for light to enter the space.

GPT-4o
S1) The image depicts the interior of a small bus or shuttle, showcasing rows of colorful, patterned seats. Large windows on the sides provide natural light, creating a bright and vibrant atmosphere.
S2) The image depicts the interior of a bus with bright rows of colorful, patterned seats. The windows on the sides lend a bright and airy feel to the transportation environment.
S3) The image shows the interior of a small passenger bus or shuttle with multiple seats arranged in rows. The seats are upholstered with a patterned fabric featuring blue and orange designs, and windows line the sides of the vehicle.
S4) The image shows the interior of a bus with rows of seats facing each other. There are windows on either side of the bus and the ceiling is white with a vent in the middle.

Figure 9: Qualitative Results for the Image Restyling Task.
Prompt: Please describe the place in the image in two sentences.
GT: office

<table>
  <tr>
    <th></th>
    <th>BLIP-2</th>
    <th>LLaVa 1.5M (Improved)</th>
    <th>BLIP-2 (Improved)</th>
  </tr>
  <tr>
    <td>S1) a painting of an office with a desk and a chair</td>
    <td>Acc: 100<br>Sr: 54.9<br>Con: 16.7<br>Sc: 58.7</td>
    <td>S1) living room<br>Acc: 0<br>Sr: 49.3<br>Con: 100<br>Sc: 100</td>
    <td>S1) office<br>Acc: 100<br>Sr: 100<br>Con: 100<br>Sc: 100</td>
  </tr>
  <tr>
    <td>S2) an office with a large window</td>
    <td></td>
    <td>S2) living room<br>Acc: 0<br>Sr: 49.3<br>Con: 100<br>Sc: 100</td>
    <td>S2) office<br>Acc: 100<br>Sr: 100<br>Con: 100<br>Sc: 100</td>
  </tr>
  <tr>
    <td>S3) an office with a large window</td>
    <td></td>
    <td>S3) living room<br>Acc: 0<br>Sr: 49.3<br>Con: 100<br>Sc: 100</td>
    <td>S3) office<br>Acc: 100<br>Sr: 100<br>Con: 100<br>Sc: 100</td>
  </tr>
  <tr>
    <td>S4) this is a black and white photo of an office</td>
    <td></td>
    <td>S4) living room<br>Acc: 0<br>Sr: 49.3<br>Con: 100<br>Sc: 100</td>
    <td>S4) office<br>Acc: 100<br>Sr: 100<br>Con: 100<br>Sc: 100</td>
  </tr>
</table>

LLaVa 1.5M
S1) The image features a living room with a red desk, a chair, and a potted plant.
S2) The image features a living room with a couch, a desk, and a window.
S3) The image features a large office space with a desk, chair, and a potted plant.
S4) The image shows a large office space with a desk, chair, and computer setup.

mPlug_cw12
S1) The image features a vibrant hotel room with a red bed, a wooden chest, a couch, and a wall with a painting, giving it a colorful, unique atmosphere.
S2) The image features a room with a large window, a couch, and a chair. There is also a computer in the room.
S3) The image features a well-decorated room with a large desk and a large window, reflecting a beautiful view outside.
S4) The image features a spacious office room with large windows, a desk, a chair, a television, and a potted plant.

moellava
S1) The image shows a room with a desk, a chair, a computer, and a window. There are also potted plants and a vase in the room.
S2) The image is a colorful, artistic representation of a room with a desk, a chair, a computer, and a potted plant. The room has a window and is decorated with stained glass.
S3) The image shows a clean and organized office space with a desk, chair, and computer. The office is well-lit with natural light from two windows, and there are potted plants on the desk, adding a touch of greenery to the room.
S4) The image is a black and white photo of a clean and organized office space. The office features a desk with a chair, a computer, and a potted plant.

Qwen-VIC
S1) A table sitting in a room with two large windows.
S2) A room with a bed, a computer, and two potted plants.
S3) A room with a bed, two chairs, a desk and two computer monitors.
S4) A black and white image of a corner office with a desk and chair.

BLIP-3
S1) a painting of an office with a desk and a chair.
S2) a room with a desk and a chair.
S3) a painting of an office with a desk and a chair.
S4) a black and white photo of an office with a desk, chair, and a computer.

GPT-4V
S1) The image depicts an interior space with a modern, cozy ambiance, featuring a red sofa and a cushioned armchair that invite relaxation. Large windows let in ample natural light, illuminating the room and a variety of indoor plants that add a touch of greenery to the space.
S2) The image appears to be an artistic, abstract representation of a bedroom with a bed in the center, a large window providing a view of the outside, and possibly a nightstand or desk with a laptop. The image is stylized with distinct, bold lines and a vibrant color palette, creating a dreamlike perspective, giving it a dreamlike or surreal quality.
S3) The image features an interior space with large windows, including large glass windows that provide a view of the exterior environment. The room is furnished with a desk on which there is a computer, a plant, and various other items, suggesting a workspace or office setting.
S4) This is a well-furnished office with a modern aesthetic, featuring a large desk, comfortable chair, and multiple computer monitors, indicative of a technology-focused workspace. Large windows provide a view of the exterior, suggesting the office is located in a high-rise building, and the presence of plants adds a touch of nature to the room.

S1) The image appears to be an office space with large windows, featuring two desks equipped with computers and an ergonomic chair. There are potted plants adding a touch of greenery, contributing to a vibrant and professional atmosphere.
S2) The image depicts a vibrant and artistic rendition of an office space, complete with desks, computers, and large windows that let in ample natural light. The stained-glass effect adds a colorful and abstract element to the scene, enhancing the overall ambiance.
S3) The image depicts a modern office space with large windows that provide a view of the outdoors. The office is furnished with a desk, computer, chair, and houseplants, creating a professional yet comfortable environment.
S4) This is a spacious office with large windows overlooking a cityscape. It has a large desk, two computer monitors, a comfy chair, and a potted plant.

Figure 10: Qualitative Results for the Image Restyling Task.
Prompt: What kind of object is in the masked region?
GT: orange

<table>
  <tr>
    <th></th>
    <th>BLIP-2</th>
    <th>LLaVa 1.5M (Improved)</th>
    <th>BLIP-2 (Improved)</th>
  </tr>
  <tr>
    <td>A1) oranges</td>
    <td>Acc: 100<br>Sat: 81.9<br>Con: 100<br>Sc: 100</td>
    <td>A1) orange<br>A2) orange<br>A3) orange</td>
    <td>A1) orange<br>A2) orange<br>A3) orange</td>
  </tr>
  <tr>
    <td>M1</td>
    <td colspan="3">
      <b>LLaVa 1.5M</b><br>
      A1) There is an orange in the masked region.<br>
      A2) There is a square object in the masked region.<br>
      A3) There is a green square in the masked region.<br>
      <b>mPlug_owl2</b><br>
      A1) In the masked region, there is an orange.<br>
      A2) The masked region is a square.<br>
      A3) The masked region contains green leaves or branches.<br>
      <b>moellava</b><br>
      A1) There is an orange in the masked region of the image.<br>
      A2) The masked region features an orange tree with oranges on it.<br>
      A3) There is a green object in the masked region, which is a green box.<br>
      <b>Owen-VL</b><br>
      A1) The masked region contains an orange fruit hanging from a tree.<br>
      A2) The object in the masked region is an orange.<br>
      A3) The object in the masked region is a green square.<br>
      <b>BLIP-3</b><br>
      A1) The masked region contains an orange.<br>
      A2) The masked region is a white rectangular shape with horizontal lines.<br>
      A3) The masked region is a green rectangle with horizontal lines.<br>
      <b>Gemini</b><br>
      A1) The masked region contains an orange.<br>
      A2) The masked region contains a fruit. It is likely a citrus fruit, given the context of the image.<br>
      A3) The masked region contains a fruit. It is likely an orange or a tangerine.<br>
      <b>GPT-4V</b><br>
      A1) I can't identify the object in the masked region of the image. However, the visible parts suggest that the surroundings include plants and possibly some fruit.<br>
      A2) I'm unable to see the masked region, so I can't determine what object it might contain. If you can describe the image or provide more context, I may be able to help!<br>
      A3) I can't see the masked region in the image you provided, so I'm unable to determine what kind of object is there. If you can describe the object or provide more context, I can help you with information related to it!<br>
      <b>GPT-4o</b><br>
      A1) The masked region in the image obscures part of a fruit, specifically an orange, growing on a tree. The surrounding branches and leaves, along with the color and texture of the visible parts, suggest that the object is an orange.<br>
      A2) The masked region appears to be obscuring parts of a tree or plant. Given the context of the surrounding branches and leaves, it is likely that the object in the masked region could be fruit or flowers growing on the tree. The background outside the masked area reveals what seems to be foliage and branches, suggesting they could be oranges. Therefore, the object in the masked region is likely a fruit, specifically an orange.<br>
      A3) The masked region of the image is likely containing a fruit, most probably an orange or a similar citrus fruit, as suggested by the context of the surrounding branches and leaves. The surrounding foliage and the small visible parts of an orange fruit further hint towards this.
    </td>
  </tr>
  <tr>
    <td>M2</td>
    <td colspan="3">
      <b>Acc: 33.3<br>Sat: 42.5<br>Con: 0<br>Sc: 47.4</b><br>
      <b>Acc: 33.3<br>Sat: 36.4<br>Con: 0<br>Sc: 30.2</b><br>
      <b>Acc: 66.7<br>Sat: 53.7<br>Con: 0<br>Sc: 40.3</b><br>
      <b>Acc: 66.7<br>Sat: 50.0<br>Con: 0<br>Sc: 50.1</b><br>
      <b>Acc: 33.3<br>Sat: 25.7<br>Con: 33.3<br>Sc: 65.2</b><br>
      <b>Acc: 66.7<br>Sat: 51.9<br>Con: 66.7<br>Sc: 77.0</b><br>
      <b>Acc: 0<br>Sat: 21.3<br>Con: 33.3<br>Sc: 58.1</b><br>
      <b>Acc: 100<br>Sat: 47.1<br>Con: 100<br>Sc: 91.5</b>
    </td>
  </tr>
  <tr>
    <td>M3</td>
    <td colspan="3"></td>
  </tr>
</table>

Figure 11: Qualitative Results for the Context Reasoning Task.
Prompt: What kind of object is in the masked region?
GT: toilet

<table>
  <tr>
    <th></th>
    <th>BLIP-2</th>
    <th>LLaVa 1.5M (Improved)</th>
    <th>BLIP-2 (Improved)</th>
  </tr>
  <tr>
    <td>A1) a sink</td>
    <td>Acc: 0<br>Sar: 48.6<br>Con: 100<br>Sc: 100</td>
    <td>A1) toilet<br>Sar: 100<br>Con: 100<br>Sc: 100</td>
    <td>A1) toilet<br>Sar: 100<br>Con: 100<br>Sc: 100</td>
  </tr>
  <tr>
    <td>A2) a sink</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>A3) a sink</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

LLaVa 1.5M
A1) The masked region in the image contains a toilet paper dispenser.
A2) The masked region in the image contains a sink.
A3) There is a sink in the masked region.

mPlug_owl2
A1) The masked region in the image contains a picture of a toilet.
A2) In the masked region, there is a sink faucet.
A3) In the masked region of the image, there is a sink.

moellava
A1) The masked region contains a white sink.
A2) The masked region contains a toilet.
A3) The masked region contains a toilet.

Owen-VL
A1) The masked region contains a radiator.
A2) The masked region contains a toilet, which is located next to the sink in the bathroom.
A3) The masked region is a white sink, which is a part of a bathroom.

BLIP-3
A1) The masked region appears to be a toilet.
A2) The masked region obscures a toilet.
A3) The masked region is over a sink.

Gemini
A1) The masked region contains a toilet bowl.
A2) The masked region contains a toilet bowl.
A3) The masked region contains a toilet brush.

GPT-4V
A1) The masked region seems to be obscuring part of a sink or basin area. It's likely a portion of the bathroom fixtures, possibly where you would find taps or plumbing connections. However, without more specifics, it's hard to determine exact details.
A2) The masked region likely contains an object found in a restroom setting, possibly concerning sanitary fixtures. Based on the surrounding objects visible in the image, it could be a toilet or another type of sink or basin.
A3) The masked region likely contains a washbasin or sink, as suggested by the shape and position of the visible parts of the object in the image.

GPT-4o
A1) The masked region seems to be concealing an object that is directly next to a sink and appears to be part of a restroom setup. Given the context and typical restroom arrangements, it's possible that the masked object is another fixture typically found in such settings, such as a paper towel dispenser, a hand dryer, a trash bin, or part of a urinal. The exact nature of the object cannot be determined so conclusively out of the surrounding context.
A2) The masked region is likely covering a toilet bowl. The objects in the surrounding area include a urinal to the left and a sink in the center, often found together in a restroom environment with a toilet being a common nearby fixture.
A3) The masked region appears to be covering part of a toilet bowl in a bathroom.

Figure 12: Qualitative Results for the Context Reasoning Task.

Reviewer paper summaries: This paper introduces a consistency evaluation framework covering question rephrasing, image restyling, and context reasoning. Their findings reveal that higher accuracy does not necessarily imply greater consistency, according to their experiment results over multiple LVLMs. Meanwhile, this paper proposes an adapter for LVLMs to improve the consistency issues in LVLM outputs.

This paper introduces a benchmark to address the consistency robustness and trust in VLMs, focusing on three tasks: question rephrasing, image restyling, and context reasoning. The benchmark includes training and testing splits, sourced from five existing dataset. The paper evaluates several state-of-the-art VLMs on this benchmark and finds that consistency and accuracy do not always align. Additionally, this paper also proposes a strategy to minimize the inconsistency across prompts via an adapter module.

This paper identifies the importance of consistency for robust and deployable VLMs and introduces a dataset for analyzing the semantic consistency of VLM outputs. The authors construct a multimodal benchmark that includes three tasks - question rephrasing, image restyling, and context reasoning, to probe the ability of VLMs in maintaining consistency in their responses to different variations of the same question. Overall, questions and images are sourced from other datasets such as InfographicsVQA, OKVQA, Google Landmarks Dataset v2, Indoor Scene Dataset, and MSCOCO. 

For evaluation, the authors define an overall metric as the harmonic mean of correctness and consistency metrics, and apply the benchmark to six widely used VLMs. The results find that high accuracy does not necessarily equate to greater consistency. And that there is great variation in the semantic consistency in different models. 

Finally, the authors propose a lightweight adapter module to improve consistency. The adapter adds prefix embeddings to the LLM input and is the only part of the mode that is trained using the train set from the proposed dataset. The results on the test set show that for the adapted models, both the accuracy and consistency were improved, with the improvement in consistency being greater. However, there was slight degradation for a single model evaluated on external dataset, OKVQA.
Summary of strengths: 1. The paper is well-written and pleasant to read, with illustrative figures and a comprehensive analysis of experiments.
2. The experiment results are sufficient.
3. The showcases provided in the appendix are really helpful for understanding the performance of this method.

1. The paper addresses an important and relatively underexplored aspect of VLM evaluation, the consistency.    
2. The paper provides a detailed analysis of several state-of-the-art VLMs, highlighting the differences in their consistency and accuracy.    
3. The measurement does not solely rely on the accuracy, allowing the benchmark to conprehensively evaluate model abilities.
4. Apart from the benchmark, this paper also proposes a method to improve consistency via an adapter module, demonstrating its effectiveness on widely used VLMs.    
5. The authors implement validation strategies to ensure the quality of their dataset.

This paper highlights a highly relevant ability that is deficient in many VLMs. In addition to accuracy, an evaluation of semantic consistency may be important for VLMs to be more reliable and consistent in real-world applications. The proposed dataset sources questions and images from various well-established datasets and introduces variations such as rephrasing, image restyling, and masking. The added value of the augmentations serve to systematically validate the consistency of VLM models. The results show how even widely used VLMs are inconsistent in the semantic formulation of their outputs. This benchmark could be of interest and use to researchers working in consistency related topics or for general improvement of VLM outputs.
Summary of weaknesses: 1. Could the authors provide some cases when the accuracy is low and consistency is high, and cases when the accuracy is high and consistency is low (if any)? How do the four metrics work in these cases?
2. Could the authors provide some discussion and comparison of the questions/images before and after editing? Like, what's the difference in semantics before and after questions rephrasing (Question) or Imag Restyling (Image)? Is it possible to lose some key information due to this editing? I am concerned that from certain types of image editing like grayscale, some key information may be lost due to image editing, which may alter the question-answering results.
3. Does the context reasoning involve some open-ended questions? How do the authors determine the ground truth (which might be multiple with similarities) and validate the ground truth accordingly?
4. I think it might be helpful to construct the entire problem in the form of <Q, I, A>, where Q refers to the question (question rephrasing), I refers to the image (image restyling), and A refers to the answer/ground truth (context reasoning), and motivate the entire method from this angle.

1. The types of consistency is not thoroughly discussed: The paper focuses primarily on semantic consistency. Other forms of consistency, such as factual consistency and cross-modal conceptual consistency, are not explored.   
2. For the context reasoning task, authors introduce different types of mask but do not provide motivation behind the designs, which is quiten confusing.
3. The O$_{all}$ is too naive. It simply averages the four employed metrics, which decrease the comprehensiveness of the measurement.

1. While the dataset is well-constructed, I feel the proposed evaluation metrics could be improved for better analytical capacity. It is uncommon to use accuracy or similarity for open-ended questions like subtasks, Image Restyling and Context Reasoning. For these tasks perhaps metrics such as CIDEr or SPICE would be a better choice. It also seems that the consistency metrics are evaluated independently to correctness since groundtruth isn't used in the calculation. It may be more informative if the relationship between consistency and accuracy rate can be quantified, eg. high consistency in a given group of responses has a high probability of being correct. 

2. The task of Image Restyling task does not seem like a common use-case in VLMs and an argument can be made that the different styles should ideally lead to different responses. Perhaps a better approach could be to edit the image such as changing the colors of items, or adding distractors such as different backgrounds, or surrounding objects or people.
Comments, suggestions and typos: N/A

**Questions:**
1. See weaknesses
2. What are the differences between S$_{GT}$ and S$_c$? Explain how they differ conceptually and numerically. It is better to see some real cases.
3. Does the proposed method deal with different masks differently? As discussed in Appendix A.11, there is performance bias towards different colors and shapes of mask, and I suppose that the corresponding method should be able to handle these.
4. What is the relationship among the four metrics? They should be corelated to each other, and a visualization could help. (This question can be replied together with weaknesses#3)
5. Why do some models exhibit low accuracy but high consistency? Such as the results of BLIP-2 in Table 2, with ACC 16.8 and Con 100.

- Final sentence in footnote 1 on p1 is confusing. 
- It is not clear how images were selected from the datasets. Were they randomly sampled?
- I assume pairwise scores in the consistency metrics refer to pairings of different variations for each image sample, ie. Rephrased 1 and Rephrased 2. This can be made clearer in the text.
- (Table 2) What is the scale for similarity scores? Is it normalized to 0~100?
- (Table 2) It is not clear what the difference between Sampling and All is. What sources of stochasticity are there if temperature is set to 0?
Overall assessment scores: 3

3

4
Metareview score: 4
Metareview: This paper presents a consistency evaluation framework covering question rephrasing, image restyling, and context reasoning. They evaluated on multiple MLLMs and show that higher accuracy does not necessarily come with higher consistency. They also propose a method to mitigate the consistency issues.
Summary of reasons to publish: The consistency issue is really novel and interesting. The findings are useful and they also proposed a solution. This is a good paper and I strongly vote for its acceptance to main conference.