Automatically Generating Visual Hallucination Test Cases for Multimodal Large Language Models

## Abstract

Abstract Visual hallucination (VH) occurs when a multimodal large language model (MLLM) generates responses with incorrect visual details for prompts. Existing methods for generating VH test cases primarily rely on human annotations, typically in the form of triples: (image, question, answer). In this paper, we introduce VHExpansion , the first automated method for expanding VH test cases for MLLMs. Given an initial VH test case, VHExpansion automatically expands it by perturbing the question and answer through negation as well as modifying the image using both common and adversarial perturbations . Additionally, we propose a new evaluation metric, symmetric accuracy , which measures the proportion of correctly answered VH test-case pairs. Each pair consists of a test case and its negated counterpart. Our theoretical analysis shows that symmetric accuracy is an unbiased evaluation metric that remains unaffected by the imbalance of VH testing cases with varying answers when an MLLM is randomly guessing the answers, whereas traditional accuracy is prone to such imbalance. We apply VHExpansion to expand three VH datasets annotated manually and use these expanded datasets to benchmark seven MLLMs. Our evaluation shows that VHExpansion effectively identifies more VH test cases. Moreover, symmetric accuracy, being unbiased, leads to different conclusions about the vulnerability of MLLMs to VH compared to traditional accuracy metric. Finally, we show that fine-tuning MLLMs on the expanded VH dataset generated by VHExpansion mitigates VH more effectively than fine-tuning on the original, manually annotated dataset. Our code is available at: https://github.com/lycheeefish/VHExpansion .

## 1 Introduction

Given a prompt containing both an image and a question, multimodal large language models (MLLMs) [, , , , , ] generate a text response. MLLMs extend the capabilities of large language models (LLMs) [, , , , ] by enabling them to understand visual inputs. An MLLM typically comprises three main components: *a vision encoder*, *a vision-language connector*, and *an LLM*. Specifically, the vision encoder extracts visual embedding vectors from the image in the prompt, while the vision-language connector aligns these visual embedding vectors with the token-based input used by the LLM. The LLM then generates the text response based on the outputs of the vision-language connector and the text in the prompt. This integration allows MLLMs to tackle complex tasks like Visual Question Answering (VQA) [, , , ].

Figure: Figure 1: An example of visual hallucination (VH) in MLLM. The red text indicates the hallucinated response, since there are actually six spots on the butterfly’s wings.
Refer to caption: extracted/5927337/intro.png

Despite significant advancements, MLLMs are prone to a critical flaw known as visual hallucination (VH) [, ], where the model generates responses containing incorrect or misleading visual information. For example, Figure 1 illustrates a VH case where the MLLM provides an incorrect response regarding the number of spots on a butterfly’s wings. VH can lead to catastrophic outcomes, especially in high-stakes applications such as autonomous driving [, ], medical diagnostics [], and content moderation []. Therefore, VH poses significant obstacles to the safe deployments of MLLMs. This concern is highlighted in the U.S. Executive Order on Trustworthy AI [], which emphasized rigorous testing of AI systems to identify and mitigate their potential harms. Therefore, developing methods to test and mitigate VH in MLLMs is crucial for ensuring their safety.

Existing VH testing relies on either manual [] or semi-automated [, ] methods to construct test cases, both of which require extensive human annotations. As MLLMs evolve rapidly, these methods struggle to scale up VH testing, limiting the number of test cases and thus hindering comprehensive testing of MLLMs’ vulnerability to VH. Furthermore, existing VH testing methods do not consider adversarial testing [, , ] in a white-box setting, where an adversary with full knowledge of the target MLLM can craft adversarial examples to trigger VH through adding human-imperceptible perturbations to the images. This is particularly relevant for open-sourced MLLMs whose model parameters are public. Thus, automated and adversarial methods for generating VH test cases are urgently needed.

Our work:  To address these challenges, we introduce VHExpansion, the first *automated* framework to generate VH test cases for MLLMs. Given an initial VH test case, VHExpansion generates additional ones using a combination of *negation* as well as *common* and *adversarial image perturbations*. Each VH test case is a VQA triple consisting of an image, a question, and a ground-truth answer. Negation flips the question and answer.
To automate negation, we leverage an LLM with a specifically designed prompt.
For common image perturbations, we process the image via frequently encountered image processing operations such as JPEG compression, Gaussian noise, etc.. For adversarial image perturbations, we add a human-imperceptible perturbation to the image so that the resulting embedding vector, generated by the vision encoder, differs significantly from the original. We formulate finding the perturbation as a constrained optimization problem, solved using Projected Gradient Descent [] or the iterative Fast Gradient Sign Method []. We apply VHExpansion to expand three existing VH datasets annotated manually and use these expanded datasets to benchmark seven MLLMs. Our evaluation demonstrates that VHExpansion effectively identifies more VH test cases.

Moreover, we introduce a new evaluation metric called *symmetric accuracy*, which measures the proportion of correctly answered VH test-case pairs, where each pair includes a VH test case and its negated counterpart. Symmetric accuracy captures the consistency of an MLLM in accurately answering both the original and negated questions.
In fact, we theoretically show that symmetric accuracy is an *unbiased evaluation metric* that remains unaffected by the imbalance of VH testing cases with varying answers when an MLLM is randomly guessing the answers, whereas traditional accuracy is prone to such imbalance. Our empirical benchmark results show that symmetric accuracy and traditional accuracy can lead to different conclusions about MLLMs’ vulnerability to VH.
For instance, on the POPE dataset [], Cambrian-1 [] achieves a higher traditional accuracy than LLaVA-NeXT [] (0.887 vs. 0.879), but performs worse in symmetric accuracy (0.745 vs. 0.798).

Finally, we demonstrate that fine-tuning an MLLM on the expanded VH test cases generated by VHExpansion significantly mitigates visual hallucinations. For example, our experiments show that when fine-tuning LLaVA-1.5 on the POPE dataset, using randomly sampled 200 VH test cases results in a symmetric accuracy of 0.180, whereas fine-tuning on both the sampled VH test cases and the corresponding expanded 1,800 more VH test cases achieves a symmetric accuracy of 0.711. This highlights the effectiveness of VHExpansion in mitigating VH in MLLMs. Additionally, our evaluation shows that fine-tuning does not compromise the model’s performance on other general-purpose VQA datasets, preserving its broader functionality.

To summarize, we make the following contributions in this work:

- •
We introduce VHExpansion, the first automated framework for generating VH test cases in MLLMs, combining negation and common and adversarial image perturbations.
- •
We propose a new evaluation metric, symmetric accuracy, to quantify an MLLM’s performance. Symmetric accuracy is unaffected by the imbalance of test cases when an MLLM makes random guessing.
- •
We demonstrate that fine-tuning MLLMs on the expanded test cases generated by VHExpansion significantly mitigates VH while maintaining general performance on other VQA datasets.

## 2 Related Work

### 2.1 MLLMs

MLLMs [, , , , , ] have revolutionized the ability of LLMs to respond to prompts containing images and questions. Recall that MLLMs typically comprise three components: a vision encoder, a vision-language connector, and an LLM. Vision encoders are often pre-trained via self-supervised learning [, ] on large datasets of unlabeled images or image-text pairs. Among the widely used vision encoders are those from the CLIP family [], including CLIP-ViT-L/14 [], EVA-CLIP ViT-g/14 [], and OpenCLIP ConvNeXt-XXL [, ]. Cambrian-1 [] also incorporates other vision encoders, including DINOv2 ViT-L/14 [] and SigLIP ViT-SO400M/14 []. Recently, several types of vision-language connectors have been introduced, such as 2-layer multilayer perceptrons (MLPs), Q-Former [], 1-layer cross-attention mechanisms [], and Spatial Visual Aggregator []. The backbone LLMs used in MLLMs can be models like Llama2 [], Llama3 [], Vicuna [], and Qwen [].

### 2.2 Methods to Generate VH Test Cases

To detect and mitigate VH in MLLMs, several methods to generate VH test cases [, , , ] have been proposed. These methods can be categorized into two types: *manual* and *semi-automatic*. Manual methods [, ] involve creating each VH test case through human effort. For example, POPE [] requires human annotation for each image to identify the objects within it and then design corresponding questions based on these objects, including some randomly introduced non-existent objects. Note that the images in POPE are also human-created.

To reduce the human labor involved in generating VH test cases, semi-automatic methods [] have been developed. For instance, VHTest uses GPT-4V [] and DALL$\cdot$E-3 [] to facilitate construction of VH test cases. Specifically, it first employs CLIP [] and DINO [] to detect images from benchmark datasets that may trigger VH in MLLMs. These images are then passed to GPT-4V to generate textual descriptions. The generated text descriptions are subsequently passed to DALL$\cdot$E-3 to create more images. Based on these AI-generated images, human workers manually identify objects within them and design questions, along with the corresponding ground-truth answers. Similarly, MMVP [] uses CLIP and DINO to identify image pairs that have a high CLIP score but a low DINO score. Human workers then manually examine the differences between these paired images and formulate questions/answers based on those differences.

### 2.3 Mitigating VH via Fine-tuning

With VH datasets constructed by these methods, MLLMs can be fine-tuned on them to mitigate VH []. This approach enables the MLLMs to learn from instances of VH, allowing them to distinguish between accurate visual representations and hallucinated content. By exposing the models to diverse VH instances during fine-tuning, they can better generalize and reduce the occurrence of hallucinations [].

## 3 Our VHExpansion

Figure 2 shows an overview of our VHExpansion. Given an initial VH test case, VHExpansion automatically generates additional VH test cases by modifying the question and answer through negation, as well as modifying the image through common and adversarial image perturbations. We denote a VH test case as $\{x_{I},x_{Q},y_{A}\}$, where $x_{I}$ and $x_{Q}$ are respectively the image and text question in the prompt, while $y_{A}$ is the ground-truth answer. To support automated evaluation, we focus on binary questions in this work, i.e., $y_{A}$ is either “yes” or “no”. Note that non-binary question-answer pairs $(x_{Q},y_{A})$ can be rewritten as binary counterparts.

Figure: Figure 2: Overview of VHExpansion. Green text and boxes indicate text and images modified by VHExpansion.
Refer to caption: extracted/5927337/overview.png

### 3.1 Modifying Question x Q subscript 𝑥 𝑄 x_{Q} italic_x start_POSTSUBSCRIPT italic_Q end_POSTSUBSCRIPT and Answer y A subscript 𝑦 𝐴 y_{A} italic_y start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT via Negation

Given a VH test case $\{x_{I},x_{Q},y_{A}\}$, the goal of negation is to transform it into $\{x_{I},\neg x_{Q},\neg y_{A}\}$. Our VHExpansion automates this process using an LLM with a custom prompt (showed in Figure 3). This prompt takes $x_{Q}$ as input and instructs the LLM to output a negated question using predefined transformation rules, such as adding negation prefixes or modifying key words to reverse the meaning of $x_{Q}$.

Figure: Figure 3: Prompt used to instruct an LLM to negate a question $x_{Q}$.

The primary intuition behind negation is that an MLLM may simply guess the answer (i.e., “yes” or “no”) correctly for binary questions without really understanding the image. In particular, some MLLMs such as LLaVA-1.5 tend to answer “yes” for binary questions []. Therefore, if the VH test cases are imbalanced and a majority of them have “yes” as ground-truth answers, such MLLMs would have high *accuracy* without understanding the images, misleading developers to think that the MLLMs are not vulnerable to visual hallucination. However, such MLLMs would be likely to answer incorrectly for the negated questions, leading to low accuracy on them. Thus, the VH test cases and their negated versions can better quantify the vulnerability of an MLLM to visual hallucination. In fact, in Section 4, we propose a new evaluation metric, called symmetric accuracy, which measures the percentage of correctly answered VH test-case pairs, each of which includes a test case and its negated version. In Section 4, we theoretically show that symmetric accuracy is unaffected by the imbalance of VH test cases with answers “yes” and “no” when the MLLM makes random guessing, while accuracy on the original VH test cases alone is prone to such imbalance.

### 3.2 Modifying Image X I subscript 𝑋 𝐼 X_{I} italic_X start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT

Common image perturbations:  In real-world scenarios, images often undergo standard editing operations for various purposes. For example, images are frequently compressed using formats like JPEG to reduce transmission costs over the Internet. These image edits are known as *common image perturbations* []. Our VHExpansion uses these perturbations to generate additional VH test cases. Given a VH test case $\{x_{I},x_{Q},y_{A}\}$, we apply a common perturbation method $\mathcal{T}$ to the image $x_{I}$, creating a new VH test case $\{\mathcal{T}(x_{I}),x_{Q},y_{A}\}$. The intuition is that for a slightly perturbed image $\mathcal{T}(x_{I})$, the ground-truth answer $y_{A}$ should remain unchanged for the same question $x_{Q}$. However, this subtle alteration may trigger VH in an MLLM. We focus on four common image perturbations: Gaussian Noise, Brightness Adjustments, Defocus Blur, and JPEG Compression. Further details on these common perturbations are provided in Section C of the Appendix.

Adversarial image perturbations:  In the context of adversarial image perturbations, we consider a white-box setting where an adversary, with full knowledge of the target MLLM’s model parameters, crafts nearly-imperceptible adversarial perturbations to generate VH test cases. Given an original VH test case $\{x_{I},x_{Q},y_{A}\}$, the adversarial image perturbation generates a new test case $\{x_{I}+\delta^{*},x_{Q},y_{A}\}$, where $\delta^{*}$ is the adversarial perturbation. Our intuition is that for a VH test case that does not trigger VH in an MLLM $M$, VHExpansion creates perturbations that cause the projected visual embedding vector from the vision-language connector to differ from the original. Conversely, if the test case already triggers VH in $M$, VHExpansion generates perturbations that make the projected visual embedding vector similar to the original.
Formally, for an MLLM $M$ with vision encoder $M_{E}$ and vision-language connector $M_{C}$, we formulate finding $\delta^{*}$ as the solution to the following constrained optimization problem:

$$ $\displaystyle\delta^{*}=$ $\displaystyle\begin{cases}\operatorname*{argmin}_{\delta}\left(-\cos\left(M_{E }\circ M_{C}(x_{I}),M_{E}\circ M_{C}(x_{I}+\delta)\right)\right),&\text{if }x_ {I}\text{ does not trigger VH},\\ \operatorname*{argmin}_{\delta}\left(\cos\left(M_{E}\circ M_{C}(x_{I}),M_{E} \circ M_{C}(x_{I}+\delta)\right)\right),&\text{if }x_{I}\text{ triggers VH}, \end{cases}$ $\displaystyle\text{s.t.}\quad||\delta||_{\infty}\leq\epsilon,$ (1) $$

where $M_{E}\circ M_{C}$ denotes the concatenation of the vision encoder and the vision-language connector, $\cos$ denotes cosine similarity, and $\epsilon$ is the $\ell_{\infty}$-norm constraint on the perturbation $\delta$ added to the image $x_{I}$. Note that when the VH test case already triggers VH, we initialize $\delta$ to be a non-zero vector with random value and apply early stopping to avoid the optimization result to be identical with the original image input $x_{I}$; and when the VH test case does not trigger VH, we initialize $\delta$ to be zero.
Our algorithm solves the optimization problem in Equation 3.2 using either Projected Gradient Descent (PGD) [] or the iterative Fast Gradient Sign Method (I-FGSM) []. PGD iteratively updates $\delta$ via gradient ascent: $\delta=\delta-\gamma\cdot\nabla_{\delta}l$, where $l=\cos\left(M_{E}\circ M_{C}(x_{I}),M_{E}\circ M_{C}(x_{I}+\delta)\right)$, followed by projecting $\delta$ onto the feasible region using $\delta=\text{clip}(\delta,-\epsilon,\epsilon)$. I-FGSM differs from PGD by using the sign of the gradient instead: $\delta=\delta-\gamma\cdot\text{sign}(\nabla_{\delta}l)$.

## 4 Theoretical Analysis

In this section, we theoretically analyze the standard accuracy metric and our proposed symmetric accuracy metric for evaluating an MLLM model’s performance when the model is making random guessing. Suppose we are given a VH test case $t=\{x_{I},x_{Q},y_{A}\}$, sampled from the distribution $\mathcal{T}$ of VH test cases, i.e., $t\sim\mathcal{T}$. Our analysis focuses on binary questions, i.e., $y_{A}$ is either “yes” or “no”. Specifically, we denote by $q$ the probability that a randomly sampled $t$ has a ground-truth answer “yes”. In other words, a randomly sampled $t$ has a ground-truth answer “no” with probability $1-q$. $q$ quantifies the imbalance of the VH test cases with answers “yes” and “no”.

We denote by $f$ an MLLM model and $f(x_{I},x_{Q})$ the MLLM’s answer for the VH test case. $f(x_{I},x_{Q})\neq y_{A}$ indicates that the MLLM hallucinates. When the MLLM model makes random guessing to answer the test case without understanding the image $x_{I}$ and question $x_{Q}$, it outputs an answer “yes” or “no” randomly. Suppose the MLLM model guesses “yes” with probability $p$ and “no” with probability $1-p$.

An evaluation metric measures the performance of an MLLM model $f$ on the VH test cases whose distribution is $\mathcal{T}$. Specifically, an evaluation metric takes $\mathcal{T}$ and $f$ as input and outputs a number (e.g., between 0 and 1), with a smaller number indicating that $f$ is more vulnerable to VH test cases from the distribution $\mathcal{T}$. An evaluation metric is *unbiased* if it does not depend on the imbalance of the VH test cases when the model $f$ makes random guessing, i.e., it does not depend on $q$. Otherwise, the evaluation metric is biased. Formally, we have the following definition.

###### Definition 1 (Unbiased Evaluation Metric) .

An evaluation metric is said to be unbiased if does not depend on $q$ when the MLLM model makes random guessing.

Next, we formally define accuracy and prove that accuracy is a biased evaluation metric.

###### Definition 2 (Accuracy) .

Accuracy is the probability that an MLLM model $f$ correctly answers a VH test case $t=\{x_{I},x_{Q},y_{A}\}$ sampled from $\mathcal{T}$. Formally, we have: $\text{accuracy}=\text{Pr}_{t\sim\mathcal{T}}(f(x_{I},x_{Q})=y_{A})$.

###### Theorem 1 .

Accuracy is a biased evaluation metric when $p\neq\frac{1}{2}$, where $p$ is the probability that the MLLM model guesses answer “yes”.

###### Proof.

Please refer to Section A in Appendix.
∎

The above theorem shows that accuracy of an MLLM model depends on $q$ once it does not guess uniformly at random, and thus can be artificially inflated by random guessing, leading to misleading conclusions on an MLLM’s vulnerability to visual hallucination.
To address this limitation, we propose a new metric called symmetric accuracy. Formally, it is defined as follows:

###### Definition 3 (Symmetric Accuracy) .

Symmetric accuracy is the probability that an MLLM model $f$ correctly answers a VH test case $t=\{x_{I},x_{Q},y_{A}\}$ sampled from $\mathcal{T}$ and its negated version. Formally, we have: symmetric $\text{accuracy}=\text{Pr}_{t\sim\mathcal{T}}(f(x_{I},x_{Q})=y_{A}\wedge f(x_{I
},\neg x_{Q})=\neg y_{A})$.

We prove that symmetric accuracy is an unbiased evaluation metric in the following theorem:

###### Theorem 2 .

Symmetric accuracy is an unbiased evaluation metric.

###### Proof.

Please refer to Section B in Appendix.
∎

###### Definition 1 (Unbiased Evaluation Metric) .

###### Definition 2 (Accuracy) .

###### Theorem 1 .

###### Proof.

###### Definition 3 (Symmetric Accuracy) .

###### Theorem 2 .

###### Proof.

## 5 Experiments

### 5.1 Experimental Setup

**Table 1: Statistics of existing VH datasets manually annotated.**
| Dataset | # Images | # VH Test Cases |
| --- | --- | --- |
| MMVP [] | 300 | 300 |
| VHTest [] | 650 | 1,200 |
| POPE [] | 500 | 9,000 |

VH datasets:
We use three popular VH datasets: MMVP [], VHTest [], and POPE []. MMVP and VHTest consist of VH test cases across various object properties in images, such as color, counting, and position. In contrast, POPE focuses on VQA test cases related to existence VH, specifically identifying whether an object is present in an image. Table 1 summarizes the key statistics of these datasets. Note that for VHTest and POPE, a single image can be used in multiple VH test cases.

MLLMs:
In our experiments, we evaluate seven MLLMs in total. In particular, six of these models are open-source, including LLaVA-1.5 [], InstructBLIP [], Qwen-VL-Chat [], LLaVA-NeXT [], LLaVA-OneVision [], and Cambrian-1 [], alongside one closed-source model, GPT-4o []. These MLLMs demonstrate state-of-the-art performance across various VQA benchmarks and have diverse model architectures. Table 2 shows details of these MLLMs.

**Table 2: Details of MLLMs.**
| MLLM | Vision Encoder | Connector | LLM |
| --- | --- | --- | --- |
| LLaVA-1.5 [] | CLIP-ViT-L/14 [] | 2-layer MLP | Llama2-7B [] |
| InstructBLIP [] | EVA-CLIP ViT-g/14 [] | Q-Former [] | Vicuna-7B [] |
| Qwen-VL-Chat [] | OpenCLIP ViT-bigG [] | 1-layer Cross-Attention | Qwen-7B [] |
| LLaVA-NEXT [] | CLIP-ViT-L/14 | 2-layer MLP | Llama3-8B [] |
| LLaVA-OneVision [] | SigLIP ViT-SO400M/14 [] | 2-layer MLP | Qwen2-7B [] |
| Cambrian-1 [] | CLIP ViT-L/14 | Spatial Visual<br>Aggregator<br>[] | Llama3-8B |
| SigLIP ViT-SO400M/14 |  |  |  |
| OpenCLIP ConvNeXt-XXL [] |  |  |  |
| DINOv2 ViT-L/14 [] |  |  |  |
| GPT-4o [] | - | - | - |

Evaluation metrics:
We use accuracy and symmetric accuracy as our evaluation metrics, both of which are formally defined in Section 4. Our theoretical analysis demonstrates that symmetric accuracy is an unbiased evaluation metric, whereas traditional accuracy is biased. In our experiments, we illustrate how symmetric accuracy leads to different conclusions about the vulnerability of MLLMs to VH compared to the traditional accuracy metric. Subsequently, we use symmetric accuracy as our default evaluation metric unless otherwise mentioned. We also report the number of successful VH test cases generated by our VHExpansion.

**Table 3: Accuracy, symmetric accuracy, and # new successful VH test cases for seven MLLMs on the three VH datasets.**
| MLLM | LLaVA-1.5 | InstructBLIP | Qwen-VL-Chat | Cambrian-1 | LLaVA-NeXT | LLaVA-OneVision | GPT-4o |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy | 0.638 | 0.533 | 0.607 | 0.649 | 0.697 | 0.717 | 0.813 |
| Symmetric<br>Accuracy | 0.356 | 0.320 | 0.210 | 0.268 | 0.430 | 0.333 | 0.663 |
| # New Successful<br>VH test cases | 145 | 166 | 175 | 178 | 126 | 152 | 85 |

Parameter settings:
Unless otherwise mentioned, we use LLaVA-1.5 on MMVP dataset by default. We use GPT-4o as the LLM to negate all questions in VH test cases due to its state-of-the-art performance. We use the default parameter settings for all MLLMs. For common image perturbations, the parameters are set as follows: Gaussian Noise standard deviation $\sigma=0.08$, Brightness Hue-Saturation
-Value space constant $c=0.5$, Defocus Blur radius $r=5$ , and JPEG Compression quality factor $q=30$. More details of these common perturbations are shown in Section C in Appendix. For adversarial image perturbations, the default setting is: $\ell_{\infty}$-norm constraint $\epsilon=8/255$, with 500 epochs for non-hallucinated VH test cases and 100 epochs for hallucinated test cases. In hallucinated test cases, each pixel of the initial perturbation is set to $5/255$ or $-5/255$ uniformly at random.

### 5.2 Experimental Results

**Table 4: Symmetric accuracy and # new successful VH test cases on the three datasets of different MLLMs before and after common image perturbations. Due to API query limits, we sample 3,000 VH test cases from the POPE dataset for GPT-4o.**
| MLLM | LLaVA-1.5 | InstructBLIP | Qwen-VL-Chat | Cambrian-1 | LLaVA-NeXT | LLaVA-OneVision | GPT-4o |
| --- | --- | --- | --- | --- | --- | --- | --- |
| No Perturbation | 0.356 | 0.320 | 0.210 | 0.268 | 0.430 | 0.333 | 0.663 |
| Gaussian Noise | 0.353 | 0.187 | 0.147 | 0.213 | 0.370 | 0.317 | 0.643 |
| Brightness | 0.317 | 0.177 | 0.160 | 0.190 | 0.357 | 0.273 | 0.613 |
| Defocus Blur | 0.353 | 0.163 | 0.193 | 0.183 | 0.297 | 0.317 | 0.543 |
| JPEG Compression | 0.373 | 0.253 | 0.213 | 0.270 | 0.410 | 0.347 | 0.657 |

Symmetric accuracy v.s. accuracy:
Table 3(c) shows accuracy and symmetric accuracy of the seven MLLMs across the three datasets MMVP, VHTest, and POPE. We have three main observations.
First, symmetric accuracy reveals different conclusions about MLLM vulnerability to VH compared to traditional accuracy. For example, on the POPE dataset, Cambrian-1 has higher traditional accuracy than LLaVA-NeXT (0.887 vs. 0.879) but performs worse in symmetric accuracy (0.745 vs. 0.798). Second, when comparing symmetric accuracy across MLLMs, GPT-4o achieves the highest scores on MMVP and VHTest, particularly on MMVP with 0.663, indicating it is less prone to visual hallucinations than other models. LLaVA-OneVision scores the highest symmetric accuracy on POPE (0.843), likely due to its fine-tuning on simpler existence-based questions and possible overlap between POPE and its training data. Third, across VH datasets, all MLLMs perform worse on VHTest, with InstructBLIP scoring only 0.117. This is likely because VHTest contains VH test cases with AI-generated images and more complex questions that the models have not trained on, making it more challenging. In addition, expanding the dataset using negation allows us to generate more new VH instances, providing additional training data for fine-tuning, which leads to more robust models.

**Table 5: Symmetric accuracy and # new successful VH test cases on the three datasets of different MLLMs before and after adversarial image perturbations. We cannot perform adversarial image perturbations for Cambrian-1 because of our limited GPU memory, and we do not have results for GPT-4o because it is closed-source.**
| Perturbation | LLaVA-1.5 | InstructBLIP | Qwen-VL-Chat | LLaVA-NeXT | LLaVA-OneVision |
| --- | --- | --- | --- | --- | --- |
| No perturbation | 0.356 | 0.320 | 0.210 | 0.430 | 0.333 |
| I-FGSM | 0.051 | 0.080 | 0.027 | 0.263 | 0.297 |
| PGD | 0.094 | 0.080 | 0.051 | 0.287 | 0.283 |

Common and adversarial image perturbations generate more VH test cases:
Table 3(i) shows the symmetric accuracy on three datasets of different MLLMs before and after common image perturbations. We observe that symmetric accuracy slightly decreases after common image perturbations in most cases. This shows that most MLLMs are generally robust against common perturbations. For instance, LLaVA-NeXT’s symmetric accuracy on the VHTest dataset drops from 0.287 to 0.272 under Gaussian Noise. However, there are still some notable exceptions. For example, Defocus Blur significantly reduces LLaVA-OneVision’s accuracy on POPE, from 0.843 to 0.646; while three of four common perturbations even increase InstructBLIP’s symmetric accuracy on VHTest.

Table 3(o) shows the symmetric accuracy on the three datasets of different MLLMs before and after adversarial image perturbations. Our main observation is that adversarial perturbations cause significant drops in symmetric accuracy for all MLLMs. For example, LLaVA-1.5’s symmetric accuracy on POPE drops sharply from 0.468 to 0.017 when performing I-FGSM to craft adversarial perturbations. When comparing I-FGSM with PGD, I-FGSM consistently results in a larger decrease in accuracy, indicating it is more effective.

To conclude, MLLMs are fairly robust to common image perturbations but remain vulnerable to adversarial ones, highlighting the need for more adversarially robust training strategies. Furthermore, both common and adversarial perturbations lead to the generation of more new VH instances, with adversarial perturbations producing more VH instances than common perturbations, since MLLMs are more vulnerable to them.

Manual verification for negation:
The correctness of our proposed symmetric accuracy metric relies on the validity of the negated questions, which are generated by LLMs. Since LLMs may exhibit hallucinations, these negated questions might not always be the negated counterparts of the original questions. Thus, it is necessary to verify whether the negated questions generated by the LLM are correct.

To validate the correctness of these negated questions, we randomly sampled 200 VQA triples (100 original-negation pairs) from each of the MMVP, VHTest, and POPE datasets, which were evaluated by four independent annotators. The task of the annotators was to judge whether each negated question was the correct negation of the corresponding original question generated by the LLM.
The annotators unanimously agreed that all negated questions were correctly generated by the LLM.
This result demonstrates the reliability of the LLM in generating valid negations.

**Table 6: Ablation study on symmetric accuracy for adversarial image perturbations for LLaVA-1.5 on MMVP dataset when using I-FGSM.**
| $\epsilon$ | 4/255 | 8/255 | 12/255 | 16/255 |
| --- | --- | --- | --- | --- |
| Symmetric<br>Accuracy | 0.080 | 0.051 | 0.047 | 0.053 |

### 5.3 Ablation Study

We conduct a comprehensive ablation study on adversarial image perturbation using I-FGSM, since it is the most effective method to generate successful VH test cases in our VHExpansion.

Impact of $\ell_{\infty}$-norm constraint $\epsilon$:
Recall that I-FGSM projects the perturbation into the feasible region defined by the $\ell_{\infty}$-norm constraint $\epsilon$ at each iteration. Table 3(p) shows the effect of varying $\epsilon$ on symmetric accuracy. We observe that symmetric accuracy initially decreases and then stabilizes as the $\ell_{\infty}$-norm constraint $\epsilon$ increases. For example, at $\epsilon=4/255$, symmetric accuracy is 0.080, dropping to 0.051 at $\epsilon=8/255$, after which it converges. This trend occurs because larger perturbations changes the visual embedding vector more significantly of an image for a non-hallucinated VH test case, which is more likely to trigger VH and thereby reducing symmetric accuracy.

Impact of perturbation step size $\gamma$:
The perturbation step size $\gamma$ controls the update in every iteration of I-FGSM. Table 3(q) shows the impact of $\gamma$ on symmetric accuracy. We observe that symmetric accuracy is relatively insensitive to different small perturbation step size $\gamma$.

Impact of iterations:
Since I-FGSM solves the optimization problem in Equation 3.2 iteratively, we study the impact of the number of iterations and present the results in Table 3(r) and Table 3(s) for hallucinated and non-hallucinated VH test cases, respectively. For hallucinated VH test cases, we observe that symmetric accuracy remains consistently low as the number of iterations increases from 50 to 150. This is because I-FGSM updates the adversarial perturbations to increase the cosine similarity between the original and perturbed images for hallucinated VH test cases, maintaining the effectiveness of VH test cases. In non-hallucinated VH test cases, symmetric accuracy initially decreases and then stabilizes as the number of iterations increases from 100 to 900.

Impact of repetition of evaluation:
Due to the inherent randomness in the decoding algorithm of MLLMs, we repeat the evaluation and report the average symmetric accuracy in Table 3(t), varying the number of repetitions. We observe that symmetric accuracy remains consistent across different repetition counts, ranging from 0.040 to 0.051. This suggests that symmetric accuracy stabilizes after only a few repetitions, with even a single evaluation providing reliable results, thus avoiding unnecessary computational overhead.

Impact of MLLM’s temperature:
Temperature controls the randomness of MLLMs’ responses, with higher temperatures typically leading to more diverse outputs. Table 3(u) shows the impact of temperature on LLaVA-1.5’s symmetric accuracy. We observe a slight increase in symmetric accuracy as the temperature increases from 0 to 1, likely because the MLLM explores more diverse outputs at higher temperatures.

### 5.4 Mitigating VH via Fine-tuning

[] demonstrate that fine-tuning MLLMs on VH datasets constructed using VH test case generation methods can help mitigate VH. In this section, we compare the symmetric accuracy across three scenarios: 1) before fine-tuning, 2) fine-tuning on original VH test cases generated by other methods, and 3) fine-tuning on original VH test cases generated by other methods combined with expanded VH test cases from our VHExpansion.

Experimental settings:
We use LLaVA-1.5 as the fine-tuning MLLM. For fine-tuning on the original VH test cases generated by other methods, we randomly sample 200 VH test cases from each of the MMVP, VHTest, and POPE datasets, along with 4,000 randomly sampled VQA triples from the LLaVA-1.5 fine-tuning data []. For fine-tuning on our expanded VH test cases, we expand the previously sampled 200 VH test cases from each of the three datasets using negation and adversarial image perturbations, resulting in 800 VH test cases. To further increase data diversity, we use GPT-4o to rephrase the questions four times for each VH test case, generating four additional versions of each. Consequently, our expanded fine-tuning set contains 4,000 VH test cases and the sampled 4,000 VQA triples from the fine-tuning data of LLaVA-1.5. All remaining VH test cases from the three VH datasets, along with their adversarially perturbed versions, are used as evaluation data.

Following LLaVA-1.5 [], we fine-tune LLaVA-1.5 using LoRA [] with a learning rate of $1.8\times 10^{-6}$ for one epoch. All other parameters are set to the default fine-tuning settings of LLaVA-1.5.

Experimental results:
The comparison results of fine-tuning are shown in Table 7 and Table 8. Our findings demonstrate that fine-tuning on our expanded VH test cases significantly improves symmetric accuracy across the three VH datasets. For instance, on the POPE dataset, symmetric accuracy increases slightly from 0.180 to 0.189 after fine-tuning on the original VH test cases, but rises substantially to 0.711 after fine-tuning on our expanded VH test cases. This highlights the effectiveness of using VH test cases generated by our VHExpansion to mitigate VH in MLLMs. Moreover, Table 8 shows that fine-tuning on our expanded VH test cases maintains the model’s performance on other general-purpose VQA datasets, MME Perception and MME Recognition [].

**Table 7: Symmetric accuracy before and after fine-tuning on different image and VQA combinations.**
|  | Before Fine-tuning | After Fine-tuning on<br>Original VH Test Cases | After Fine-tuning on Our<br>Expanded VH Test Cases |
| --- | --- | --- | --- |
| MMVP | 0.207 | 0.172 | 0.343 |
| VHTest | 0.206 | 0.208 | 0.225 |
| POPE | 0.180 | 0.189 | 0.711 |

**Table 8: Scores on MME Perception and MME Cognition before and after fine-tuning.**
|  | Before Fine-tuning | After Fine-tuning on<br>Original VH Test Cases | After Fine-tuning on Our<br>Expanded VH Test Cases |
| --- | --- | --- | --- |
| MME Perception | 1459.3 | 1456.7 | 1434.4 |
| MME Cognition | 335.4 | 327.5 | 323.9 |

Figure: (a) MMVP dataset
Refer to caption: extracted/5927337/Ablation_1.png

Impact of fine-tuning learning rate:
Figure 4 illustrates the impact of different fine-tuning learning rates on symmetric accuracy for the MMVP dataset, scores on MME Perception and scores on MME Cognition. We observe that performance across these datasets is highly sensitive to the fine-tuning learning rate. At the learning rate of $1.8\times 10^{-6}$, the fine-tuned MLLM achieves the best trade-off among performances on all three datasets.

## 6 Conclusion

In this paper, we introduce VHExpansion, an automated framework to generate VH test cases for MLLMs. VHExpansion significantly advances VH testing by automating the generation of test cases through techniques such as negation and image perturbations, both common and adversarial. We also propose an unbiased evaluation metric, symmetric accuracy, to measure the consistency of MLLMs in answering VH test cases and their negated counterparts. Our experiments demonstrate that, given VH test cases, VHExpansion can find more successful VH test cases. Importantly, fine-tuning MLLMs on the expanded VH test cases generated by VHExpansion significantly mitigates VH, while maintaining general performance on standard VQA tasks.

## Appendix A Proof of Theorem 1

###### Proof.

Standard accuracy is defined as: $\text{Accuracy}=\text{Pr}_{t\sim\mathcal{T}}(f(x_{I},x_{Q})=y_{A})$. Since the model’s predictions are independent of $y_{A}$ when random guessing:

$$ $\displaystyle E[\text{Accuracy}]$ $\displaystyle=P(y_{A}=\text{Yes})\cdot P(f(x_{I},x_{Q})=\text{Yes})+P(y_{A}= \text{No})\cdot P(f(x_{I},x_{Q})=\text{No})$ $\displaystyle=q\cdot p+(1-q)\cdot(1-p)$ $\displaystyle=1+(2p-1)\cdot q-p.$ $$

This expression shows that $E[\text{Accuracy}]$ depends on the class distribution ($P(y_{A}=\text{Yes})$) if $p\neq\frac{1}{2}$. If the model’s bias aligns with the majority class (e.g., $p$ is large when $P(y_{A}=\text{Yes})$ is large), $E[\text{Accuracy}]$ is artificially inflated, even though the model is merely guessing.

Therefore, standard accuracy is biased due to class imbalance and model bias.
∎

###### Proof.

## Appendix B Proof of Theorem 2

###### Proof.

Symmetric accuracy is defined as: $\text{Symmetric Accuracy}=\text{Pr}_{t\sim\mathcal{T}}(f(x_{I},x_{Q})=y_{A}
\wedge f(x_{I},\neg x_{Q})=\neg y_{A})$. Since model predictions are independent of $y_{A}$ and independent between $x_{Q}$ and $\neg x_{Q}$ under random guessing:

$$ $\displaystyle E[\text{Symmetric Accuracy}]$ $\displaystyle=P(f(x_{I},x_{Q})=y_{A})\cdot P(f(x_{I},\neg x_{Q})=\neg y_{A})$ $\displaystyle=P(y_{A}=\text{Yes})\cdot P(f(x_{I},x_{Q})=\text{Yes})\cdot P(f(x _{I},\neg x_{Q})=\text{No})$ $\displaystyle\quad+P(y_{A}=\text{No})\cdot P(f(x_{I},x_{Q})=\text{No})\cdot P( f(x_{I},\neg x_{Q})=\text{Yes})$ $\displaystyle=q\cdot p(1-p)+(1-q)\cdot p(1-p)$ $\displaystyle=p(1-p).$ (2) $$

Therefore,$E[\text{Symmetric Accuracy}]=p(1-p)$, which is independent of the class distribution ($P(y_{A}=\text{Yes})$). Thus, symmetric accuracy is an unbiased evaluation metric with respect to class imbalance.

∎

###### Proof.

## Appendix C Details of Common Image Perturbations

- •
Gaussian Noise
In this method, Gaussian noise is randomly sampled from a distribution with zero mean and a standard deviation of $\sigma$. The image pixel values are first converted to the range [0, 1], and the generated noise is then added to these values. This process simulates the noise real-world images might experience during transmission. In our experiments, the standard deviation $\sigma$ is set to 0.08.
- •
Brightness
This method adjusts image brightness by modifying its V (value) channel in the HSV color space. The input image is first normalized to [0, 1] and converted from RGB to HSV. The brightness is then altered by adding a constant $c$ to the V channel, with values clipped to the range [0, 1]. The image is finally converted back to RGB. In our experiments, the constant $c$ is set to 0.5.
- •
Defocus Blur
This method applies a defocus blur to the image using a disk-shaped kernel. The input image is normalized to [0, 1], and a disk kernel of radius $c$ is generated. Each of the three RGB channels is filtered independently with this kernel, then recombined and clipped to the range [0, 1]. The radius $c$ is set to 5 in our experiments.
- •
JPEG Compression
This method compresses the input image using a specified quality factor $q$. Lower $q$ values result in higher compression and more artifacts, while higher values retain more image quality. In our experiments, the quality factor $q$ is set to 30.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The adversarial attack method employed appears somewhat outdated. Would incorporating more advanced techniques enhance the effectiveness of the proposed framework?
2. Could perturbations, such as defocus blur, degrade the image to a point where the question becomes unanswerable based on the available visual information?
soundness: 2
strengths: The paper is well-written, with clearly defined concepts that are easy to understand. The methodology and experimental design are presented in detail, ensuring transparency and allowing readers to follow the research process with ease.

The authors enhance the rigor of their approach by conducting manual verification to confirm the correctness of the proposed negation operation.

Thorough experiments and comprehensive analyses effectively demonstrate the utility and impact of the VHExpansion framework. Moreover, the paper highlights how the newly introduced symmetric accuracy metric offers a more robust and unbiased evaluation of MLLM performance on VH test cases. This provides insights for future researchers aiming to tackle these challenges.
summary: This paper introduces VHExpansion, an automated framework for generating additional visual hallucination (VH) test cases based on existing ones to evaluate multimodal large language models (MLLMs). Targeting binary questions, VHExpansion enhances current test cases by questions and answers through negation, and modifying images using both common and adversarial perturbations. The authors propose a new evaluation metric, symmetric accuracy, which serves as an unbiased measure of MLLM performance in addressing VH by assessing the model’s ability to correctly answer both original and negated questions. Experimental results show that VHExpansion effectively identifies more VH test cases and that fine-tuning MLLMs on the expanded dataset significantly reduces VH without compromising overall performance on other visual question answering (VQA) tasks.
weaknesses: 1. The proposed method's focus on binary questions limits its generalizability, as both the VHExpansion framework and the symmetric accuracy metric may not directly apply to other types of VH datasets with non-binary answer formats. It would be valuable to explore whether the performance improvements from fine-tuning on expanded VH test cases can transfer to other VH datasets with different answer structures, such as multiple-choice questions. Although the symmetric accuracy metric is tailored to binary questions, the image perturbation techniques could potentially be applied to other VH datasets.  
2. As indicated in line 288, the VH datasets used in this study primarily focus on low-level understanding, such as object properties (e.g., color, counting, and position) and object presence. It would be interesting to evaluate whether the proposed method could be effective for VH scenarios related to more advanced comprehension, like sentiment or context-based analysis.
3. The paper would benefit from a more detailed analysis of question types. For example, while the authors note that *three of four common perturbations even increase InstructBLIP’s symmetric accuracy on VHTest,* they do not delve into the underlying reasons. Analyzing how the proposed method performs across different question types and the specific capabilities it tests would provide more insight. For instance, a question involving counting (as shown in Fig. 1) might become more challenging with defocus blur, whereas the question in Fig. 2 might benefit from the same blur by reducing visual distractions.
4. The experimental setup in the "MITIGATING VH VIA FINE-TUNING" section (5.4), claimed as a major contribution, has some issues. The comparison between "fine-tuning on the original VH test cases (baseline)" and "fine-tuning on expanded VH test cases using VHExpansion (expanded)" is not entirely fair. In the *baseline* condition, the fine-tuning dataset includes 200 VH samples and 4,000 general VQA data points, while the *expanded* condition features 4,000 VH samples alongside 4,000 general VQA data points. This imbalance skews the comparison, as the *expanded* condition naturally emphasizes VH cases more. A more balanced comparison would involve scaling (repeating) the VH questions in *baseline* up to match the 4,000 samples in the *expanded* condition, thus equalizing the training set size and ensuring a fair evaluation.
5. Additionally, the use of symmetric accuracy in the evaluation poses a challenge. The *baseline* condition does not include negated questions, yet the evaluation includes them. A fairer approach would be to assess performance using standard accuracy or conduct an ablation study that excludes negation questions in the *expanded* condition. This would clarify whether the observed improvements in VH performance are indeed due to the proposed method.



**Minor Points:**

- In Tables 3-5, the metric *# New successful VH test cases* is used. It would be beneficial to also provide the ratio of successful cases to the total number of generated questions for easier comparisons.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 3
strengths: + The proposed test case generation method can automatically expand the number of VH data samples, reducing reliance on human annotations and enhancing scalability in testing multimodal large language models.
+ This paper considered the evaluation bias and propose an unbiased evaluation metric, offering a more reliable assessment compared to traditional accuracy metrics.
+ Experiments are extensive and sufficient.
summary: This paper introduces VHExpansion, an automated method for expanding Visual Hallucination (VH) test cases in multimodal large language models (MLLMs). VH occurs when an MLLM generates responses with incorrect visual details. The current methods for generating VH test cases are primarily dependent on human annotations. VHExpansion, however, expands these test cases by altering the question and answer through negation and modifying the image using both common and adversarial perturbations.

The paper also proposes a new evaluation metric, symmetric accuracy, which measures the proportion of correctly answered VH test-case pairs, including a test case and its negated counterpart. The authors argue that symmetric accuracy is an unbiased metric, unaffected by the imbalance of VH testing cases with varying answers when an MLLM is randomly guessing the answers, unlike traditional accuracy.

The researchers applied VHExpansion to three VH datasets and used these expanded datasets to benchmark seven MLLMs. The results showed that VHExpansion effectively identifies more VH test cases and that symmetric accuracy provides a more unbiased evaluation of MLLMs' vulnerability to VH. The paper also demonstrates that fine-tuning MLLMs on the expanded VH dataset generated by VHExpansion is more effective in mitigating VH than fine-tuning on the original, manually annotated dataset.
weaknesses: - While the paper discusses common and adversarial perturbations, it might not fully explore the spectrum of possible perturbations that could trigger visual hallucinations in MLLMs, such as resize/affine transform/crop/generation-based image editting, etc. There could be more sophisticated or subtle perturbations not covered.
- The method relies on an LLM to automate negation of questions, which might introduce its own errors or biases. The makes the method still need correction by human and thus the method is still semi-automatic instead of the stated fully-automatic.
- In my opinion, Table 8 shows that fine-tuning on expanded VH test cases degrades the model’s
performance on other general-purpose VQA datasets (MME in this case) since the scores are harmed. However, authors tend to interpret the results as " the model’s performance are maintained" in line 565.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: (1) Since the negation part modifies the question using LLM, has the quality of the modified questions been validated?
soundness: 2
strengths: (1) The authors identified a potential bias in the original yes/no response QA questions and proposed a symmetric accuracy evaluation metric, which is very insightful for current evaluations.

(2) The authors conducted comprehensive experiments, including a comparison between traditional metrics and the new metric, as well as improvements in model performance after fine-tuning.

(3) Traditional hallucination benchmarks mostly rely on manual construction, while automated generation methods greatly reduce the need for human labor.
summary: This paper introduces VHExpansion, an automated framework for generating visual hallucination (VH) test cases in multimodal large language models (MLLMs) using negation and image perturbations. It proposes a new metric, symmetric accuracy, to address bias in traditional accuracy, providing a more reliable measure of VH vulnerability. Experiments demonstrate that symmetric accuracy reveals more about MLLM susceptibility to VH, and fine-tuning with VHExpansion cases reduces hallucinations effectively.
weaknesses: (1) As an automated method for generating VH test cases, this approach addresses the issue of insufficient data volume in previous benchmarks like MMHAL-BENCH and POPE.

(2) The entire paper seems like an application of transferring an attack method to VLMs, lacking novelty, as the attack approach itself has been previously used.

(3) The overall paradigm of automatically constructing benchmarks is also very similar to many works, such as:
     1. Autobencher: Creating salient, novel, difficult datasets for language models
     2. Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model
     3. How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
For example, in the third paper, some test cases were also constructed using a similar attack method.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1, In this paper, the authors stated that the proposed metric (symmetric accuracy) is better than the standard accuracy. Other than the theoretical analysis, are there any human evaluation/human preference that can further show that this metric is better preferred?
2, In table.7, can the authors provide more explanation about why on some dataset (POPE), the improvement is huge while on others it is not as significant.
3, Other than Llava-1.5, have the authors tried fine-tuning on different open source models?(Table.7) Are there also improvements in the performances?
soundness: 2
strengths: 1, The paper has done extensive ablation study and has evaluated different hyperparameters.
2, The paper is clearly written.
summary: This paper introduces VHExpansion, an automated method for generating visual hallucination test cases for MLLMs. Each test case is generated by perturbing the question and answer through negation and modifying the image using common and adversarial perturbations. The paper also proposes an evaluation metric, named symmetric accuracy and shows it's unbiased.
weaknesses: 1, The paper mainly addresses the MLLM hallucination by constructing more data and then fine-tuning on them, which is not very exciting.
2, In Table.8, we can see the proposed method causes the MLLM comprehensive ability to decrease.