Visual Hallucinations of Multi-modal Large Language Models (2024)

Abstract: Visual hallucination (VH) means that a multimodal LLM (MLLM) imagines incorrect details about an image in visual question answering. Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs’ performance under VH due to limited diversity of such VH instances. In this work, we propose a tool called VHTest to generate a diverse set of VH instances. Specifically, VHTest finds some initial VH instances in existing image datasets (e.g., COCO), generates a text description for each VH mode, and uses a text-toimage generative model (e.g., DALL·E-3) to generate VH images based on the text descriptions. We collect a benchmark dataset with 1,200 VH instances in 8 VH modes using VHTest. We find that existing MLLMs such as GPT-4V, LLaVA-1.5, and MiniGPT-v2 hallucinate for a large fraction of the instances in our benchmark. Moreover, we find that finetuning an MLLM using our benchmark dataset reduces its likelihood to hallucinate without sacrificing its performance on other benchmarks. Our benchmarks are publicly available: https://github.com/wenhuang2000/VHTest.

heading: 1 Introduction
text: A multi-modal LLM (MLLM) (Yang et al., 2023; Zhu et al., 2023; Chen et al., 2022; Huang et al., 2023b; Tiong et al., 2022) generates a text response for a given image and question. An MLLM typically comprises three components: a vision encoder, a vision-language connector, and an LLM. The vision encoder (e.g., CLIP (Radford et al., 2021)) converts an image into an embedding vector. The vision-language connector projects an image embedding vector into the LLM’s word embedding space. The projected vector is concatenated with the token embeddings of the question to form an input to the LLM, which generates a text response.
*Equal contributions.
However, MLLMs often generate text responses containing factually incorrect details about an image, known as visual hallucination (VH) (Li et al., 2023; Liu et al., 2024b). Figure 1 shows an example where the MLLM hallucinates two lamps, contradicting the three lamps in the image. VHs in MLLMs pose obstacles to developing safe and trustworthy AI, which is emphasized in a recent U.S. Executive Order calling for rigorous testing to address potential harms from advanced AI systems (The White House, 2023).
Prior works have tried to benchmark MLLMs’ VHs related to object existence (Li et al., 2023; Liu et al., 2024a), optical character recognition (OCR), object counting, object positions comparing (Fu et al., 2023), orientation, and viewpoint (Tong et al., 2024) (concurrent to ours). However, they collect VH images only from existing image datasets like COCO (Lin et al., 2014). This limits the diversity of VH images since they can only find a limited number of them. Moreover, existing image datasets may have been used to pre-train an MLLM, leading to data contamination (Jacovi et al., 2023; Sainz et al., 2023). As a result, such VH images lead to a biased understanding of an MLLM’s performance, e.g., an MLLM is incorrectly concluded to perform well under VH.
Our Work We propose VHTest, a tool that generates VH instances through adversarial construction
9614
to expose visual hallucinations within MLLMs. A VH instance is a triple (an image, a question, a reference answer). Our VHTest has three key steps. Step I finds initial VH instances using existing image datasets like COCO. Specifically, we first identify image pairs with high CLIP embedding similarity but low DINO v2 (Oquab et al., 2023) embedding similarity. Such image pairs have contradictory similarities from two powerful vision encoders, indicating potential VHs. We note that Step I is also used in a concurrent work MMVP (Tong et al., 2024). Unlike MMVP, which directly collects initial VH instances to build their benchmarks, we treat them as the raw materials for further adversarial construction. We then manually design questions and reference answers for these images to obtain initial VH instances. Step II generates a text description for a VH mode derived from the initial VH instances. A text description describes visual properties of VH images that are likely to cause MLLMs to hallucinate. Finally, Step III uses a text-to-image generative model (e.g., DALL-E 3) to generate new images based on the text descriptions. Moreover, based on some templates, we design questions and reference answers for the generated images to construct VH instances. Using our VHTest, we construct a new benchmark dataset for evaluating VHs in MLLMs. Our benchmark contains 1,200 VH instances covering 8 VH modes. The 8 VH modes are related to existence, shape, color, orientation, OCR, size, position, and counting of visual objects in an image. Note that shape and size VH modes are formulated by us, while the other 6 VH modes were also considered in prior (Yang et al., 2023) and concurrent (Tong et al., 2024) studies. Figure 2 and Figure 7 show some VH instances generated by VHTest. We comprehensively evaluate state-of-the-art MLLMs, including GPT-4V, LLaVA-1.5, and MiniGPT-v2 on our benchmark. Our results show that MLLMs hallucinate for a large fraction of the VH instances in our benchmark. For exam-
ple, GPT-4V, LLaVA-1.5, and MiniGPT-v2 only achieve overall accuracy of 0.383, 0.229, and 0.075 on our benchmark, respectively. We also find that MLLMs have different performance across VH modes. For example, GPT-4V is most prone to orientation VH with 0.153 accuracy; while LLaVA1.5 and MiniGPT-v2 are most susceptible to OCR VH with 0.127 and 0.000 accuracy, respectively.
Finally, we show that fine-tuning an MLLM using our benchmark dataset mitigates VH. Specifically, we divide our benchmark into the training/testing splits with a ratio 80%/20%. We then fine-tune the LLaVA-1.5 model on the training split. After fine-tuning, we evaluate the model on the testing split. Our results show fine-tuning reduces the likelihood for an MLLM to hallucinate. For example, in position VH mode, the fine-tuned LLaVA-1.5 gains 0.200 accuracy from 0.333 to 0.533. Moreover, fine-tuning maintains model performance on other benchmark datasets.
2 Definitions
VH Modes We can categorize visual properties of objects in an image into individual properties, which can be attributed to individual objects (e.g., existence, shape, color, orientation, and OCR), and group properties, which emerge from comparisons across multiple objects (e.g., relative size, relative position, and counting). Based on such categorization, we have 8 VH modes as follows. In particular, each VH mode occurs when an MLLM’s text response is factually incorrect with respect to the corresponding visual property in an image.
1. Existence VH: O is the set of objects in image I , and O′ is the set of objects an MLLM identifies in I . Existence VH occurs if: ∃oi ∈ O s.t. oi /∈ O′ or ∃o′j ∈ O′ s.t. o′j /∈ O. In other words, the MLLM misses at least one object in I or fabricates at least one nonexistent object.
2. Shape VH: Let S = {s(oi)}ni=1 denote the list of shapes for objects {oi}ni=1 in I , and S′ = {s′(oi)}ni=1 is the corresponding list of shapes identified by an MLLM. A shape VH occurs if: ∃oi s.t. s(oi) ̸= s′(oi). Intuitively, the MLLM fails to accurately describe the shape of at least one object in I .
3. Color VH: Let C = {c(oi)}ni=1 denote the list of colors for objects {oi}ni=1 in I , and C ′ = {c′(oi)}ni=1 is the corresponding list
of colors identified by an MLLM. A color VH occurs if the MLLM fails to accurately identify the color of at least one object in I . Formally, we have: ∃oi s.t. c(oi) ̸= c′(oi).
4. Orientation VH: An LLM fails to precisely recognize the facing orientation of at least one object in an image.
5. OCR VH: An MLLM fails to accurately identify at least one character in an image.
6. Size VH: An MLLM fails to accurately compare the relative sizes of multiple objects in an image.
7. Position VH: An MLLM fails to accurately identify spatial relationships between objects in an image.
8. Counting VH: An MLLM exhibits a counting VH mode when it cannot accurately enumerate the number of objects in an image.
VH Instance An VH instance is a triple {xi, xt, yr}, where xi is an image, xt is a question, and yr is a reference answer. We say a VH instance succeeds for an MLLM if and only if the MLLM’s text response for xi and xt is factually incorrect compared to the reference answer yr. For instance, in the example shown in Figure 1, the reference answer is “three lamps”, while the MLLM’s text response indicates two lamps.

heading: 3 Our VHTest

heading: 3.1 Step I: Finding Initial VH Instances
text: Since the CLIP vision encoder is the backbone of many popular MLLMs such as LLaVA (Liu et al., 2023), LLaMA-Adapter (Gao et al., 2023), and mPLUG-Owl (Ye et al., 2023), we leverage CLIP to find initial VH instances. Our goal is to find images in an existing image dataset (e.g., COCO) that are incorrectly embedded by the CLIP vision encoder to have high similarity, despite differences in visual semantics. Such images may lead to VH for MLLMs due to their incorrect embeddings.
Specifically, given an image pair (x1, x2), we compute the cosine similarity between their CLIP embedding vectors, i.e., cos(fC(x1), fC(x2)), where fC(·) is an embedding vector output by CLIP. Moreover, we also use DINO v2 (Oquab et al., 2023) as a reference vision encoder to compute their embedding vectors and compute their
cosine similarity, i.e., cos(fD(x1), fD(x2)), where fD(·) is an embedding vector output by DINO v2. Then, we find image pairs that have large cosine similarity under CLIP but small cosine similarity under DINO v2. In particular, we find image pairs that satisfy cos(fC(x1), fC(x2)) ≥ 0.9 and cos(fD(x1), fD(x2)) ≤ 0.55. Such image pairs are our candidates.
Among the candidates, we further select the top 200 pairs with the largest cosine similarity under CLIP. Moreover, we manually design questions and reference answers to form 800 initial VH instances, where 100 for each VH mode and an image may be used in multiple VH instances. Finally, we test an MLLM (called testing MLLM, e.g., GPT-4V) on them. With manual verification, GPT-4V hallucinates on 204 of them (called successful initial VH instances). Table 1 shows the number of successful initial instances in each VH mode.

heading: 3.2 Step II: Text Description Generation
text: Given the initial VH instances, we use an MLLM (called description-generation MLLM, e.g., GPT4V) to generate a text description for each VH mode. The text description aims to guide a textto-image generative model (in Step III) to generate more images that are likely to trigger VH in MLLMs.
Using a Successful Initial VH Instance We first describe how to leverage prompt engineering to generate a text description based on one successful initial VH instance. Specifically, we construct an example comprised of: 1) the VH instance’s image and question, 2) the testing MLLM’s hallucinated response, and 3) the detected hallucination in the
response. We then add an additional prompt asking the description-generation MLLM to generate a text description that explains potential causes underlying the observed VH and describes how to generate more images. We show a summary version of our prompt in Figure 4 while the full prompt is shown in Figure 9 in Appendix.
Using an Unsuccessful Initial VH Instance We also design a prompt to generate a text description based on an unsuccessful initial VH instance for which the testing MLLM does not hallucinate. Such prompt is needed when not enough successful initial VH instances are available for a VH mode. Our idea is to construct a prompt with some hypothetical hallucinated detail regarding the VH image. Specifically, we construct an example comprised of: 1) the VH instance’s image and question, 2) a hypothetical hallucinated response, and 3) the detected hallucination in the hypothetical response. Given the example, we also add an additional prompt like we discussed above. We show a summary version of our prompt in Figure 5 while the full prompt is shown in Figure 10 in Appendix.
Integrating Multiple VH Instances To increase diversity of our text description, for each VH mode, we generate 10 text descriptions based on 10 successful initial VH instances. If a VH mode has less than 10 successful initial VH instances, we generate the remaining text descriptions based on unsuccessful ones. Then, for each VH mode, we use a prompt in Figure 11 in Appendix for a descriptiongeneration MLLM to summarize the 10 text descriptions as the final one. Appendix A shows our generated text description for each VH mode.

heading: 3.3 Step III: Generating More VH Instances
text: We generate more VH instances based on the text descriptions. Recall that a VH instance consists of an image, a question, and a reference answer. Therefore, we describe how to generate each component in the following.
VH Image We use a text-to-image generative model (e.g., DALL·E-3 in our experiments) to generate VH images based on the text descriptions. Specifically, to generate an image in a VH mode, we append the corresponding text description to the prompt in Figure 12 in Appendix for the textto-image generative model to generate an image.
VH Question Given a VH image, we prepare a question based on it. Specifically, we leverage object-driven templates to create diverse and relevant VH questions. For instance, “Describe the shape of the [object] in the picture.” is a template for the shape VH mode. We curate question templates for each VH mode and they are shown in Appendix D. Given a template for a VH mode, a human worker generates a question via manually analyzing the objects in the VH image. For instance, the human worker may replace the [object] as “pear” in the template above when the VH image contains a pear. The human worker verifies that the question should have a non-ambiguous an-
swer based on the VH image. If no questions with non-ambiguous answers can be constructed, the VH image is discarded.
VH Reference Answer Given a VH image and a question, a human worker also provides a factually correct answer as a reference answer via manually analyzing the VH image. The triple (image, question, reference answer) is a VH instance.

heading: 3.4 Benchmark Construction
text: We use VHTest to build two benchmarks. Specifically, we find initial VH instances in COCO (Lin et al., 2014) and we generate 150 VH instances for each VH mode, which results in 1,200 VH instances across 8 VH modes in total. This benchmark consists of “open-ended question” (OEQ), which requires manually labeling the responses of MLLMs when testing them on the benchmark. Therefore, to facilitate automatic evaluation when testing MLLMs, we also construct a closed-ended “yes/no question” (YNQ) version of the benchmark.
Specifically, for each VH instance, we convert the open-ended question into a binary “yes/no” question. For instance, the open-ended question “Describe the shape of the pear in the picture.” is converted into “Is the shape of the pear in the picture a square?”. Moreover, the reference answer is converted into yes or no. We construct the binary questions to ensure that the YNQ benchmark has a 50/50 split of “yes”/“no” reference answers. Moreover, to ensure quality of our benchmarks, we measure the inter-annotator agreement to evaluate the consistency among different human annotators, provided in Appendix B.1.
Our benchmark construction took approximately 300 human-hours in total.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: MLLMs We evaluate three state-of-the-art MLLMs on our benchmarks: GPT-4V with its “gpt4-vision-preview” version, LLaVA-1.5-13b (Liu et al., 2023), and MiniGPT-v2 (Chen et al., 2023a).
Evaluation Metric We use accuracy as an evaluation metric. Given an MLLM, we use it to produce a text response for each VH instance (more precisely, the image and question in a VH instance) in our OEQ benchmark; we manually analyze the text responses and compare with the reference answers; and accuracy is the fraction of the VH in-
stances for which the MLLM’s text responses are factually correct. For the YNQ benchmark, we can automatically calculate the accuracy of an MLLM since the reference answers are just yes or no. Note that a smaller accuracy indicates that an MLLM is more likely to hallucinate on our benchmarks, which shows that our VHTest is better at generating successful VH instances.

heading: 4.2 Testing VH in MLLMs
text: Initial VH Instances are Insufficient Table 1 shows the accuracy of GPT-4V on the 100 initial VH instances from COCO for each VH mode, and the number of successful initial VH instances in each VH mode. We observe that a large fraction of the initial VH instances are not successful. In particular, the average accuracy of GPT-4V across the 8 VH modes is 0.745, which means that only 204 of the 800 initial VH instances make GPT-4V hallucinate. These results show that VH instances in existing image datasets are insufficient at testing MLLMs. For instance, given the results in Table 1, one may conclude that GPT-4V does not hallucinate for the shape VH mode since its accuracy is 1.00 for this VH mode. However, as we will discuss in the following, GPT-4V hallucinates substantially in the shape VH mode on our benchmarks.
MLLMs Hallucinate on our VHTest Benchmarks Table 2 and Table 3 show the accuracy of GPT-4V, LLaVA-1.5, and MiniGPT-v2 on our OEQ and YNQ benchmarks, respectively. We observe that these MLLMs achieve a strikingly low
average accuracy across the eight VH modes: on average 0.229 and 0.561 for all MLLMs on OEQ and YNQ benchmarks, respectively. For example, on average, 925 out of the 1,200 VH instances in our OEQ benchmark induce these MLLMs to hallucinate. It is worth noting that random guessing can achieve an accuracy of 0.5 on our YNQ benchmark since it is a balanced yes/no benchmark. Our results indicate that VHTest is highly effective at generating successful VH instances.
Among the three MLLMs, GPT-4V and MiniGPT-v2 achieve the highest and lowest accuracy on our benchmarks, respectively. This suggests that GPT-4V is the most truthful while MiniGPT-v2 is the least truthful on our benchmarks. Furthermore, we observe that MLLMs perform the poorest on the orientation, counting, and OCR VH modes based on their average accuracy for each VH mode. For example, the average accuracy of the three MLLMs on orientation VH mode is only 0.14. This implies that orientation is the most challenging VH mode, causing more VHs in MLLMs compared to other VH modes.
We also evaluate Gemini-pro-vision (Gemini) (March 2024 version), ShareGPT4V-13b (Chen et al., 2023b), InstructBLIP-13b (Dai et al., 2023), and Qwen-VL-Chat-7b (Bai et al., 2023) on our YNQ benchmark. The results are shown in Appendix B.2. We find that these four MLLMs also suffer from visual hallucinations on our benchmark, with their accuracies all hovering around 0.54.

heading: 4.3 Ablation Study
text: Unless otherwise mentioned, we use the counting VH mode due to the GPT-4V query limitation.
Using Different Text-to-image Generative Models in Step III We use DALL·E-3 to generate VH images in Step III of our VHTest. We also evaluate other text-to-image generative models, including Midjourney 6 (Midjourney), Stable Diffusion XL 1.0 (Podell et al., 2024), and Stable Diffusion 2.1 (Rombach et al., 2022) by using each of them to generate 30 VH instances, respectively. DALL·E-3 rewrites a prompt automatically to add more details, whereas the other three models lack this capability. We find that other text-to-image generative models cannot generate high-quality images when directly using the text description from Step II as a prompt. Therefore, we use GPT-4 to generate prompts given the text description for other text-to-image generative models. Specifically, we use the prompt in Figure 13 for GPT-4 to generate 30 prompts. Then, we use each text-to-image generative model to generate 30 VH images with the generated prompts. Finally, we manually craft the questions and reference answers to form 30 VH instances. Table 4 shows the accuracy of the three MLLMs on the 30 VH instances generated by different text-to-image generative models. We observe that these MLLMs achieve the lowest average accuracy of 0.144 when using DALL·E-3 in Step III. This indicates that DALL·E-3 is the most effective tool in generating VH instances that are likely to trigger VHs in MLLMs.
Do Successful Initial VH Instances Help? We analyze the impact of successful initial VH instances on generating VH instances. Specifically, we use Step II of our VHTest to generate a text description based on three successful initial VH instances and a text description based on three unsuccessful initial VH instances in the counting VH mode. Then, we use Step III of our VHTest to generate 10 VH instances based on each text description. Table 5 shows the accuracy of the 3 MLLMs on the 10 VH instances in the two scenarios. Our results show that the VH instances generated using successful initial VH instances substantially reduce accuracy across GPT-4V, LLaVA-1.5, and MiniGPT-v2. A lower accuracy indicates a higher degree of VH in MLLMs. Our results show that using successful initial VH instances, our VHTest is more likely to generate successful VH instances.

heading: 4.4 Mitigating VH in MLLMs
text: Fine-tuning LLaVA-1.5 on our Benchmark We also study whether fine-tuning an MLLM on our benchmark makes it less likely to hallucinate. Towards this goal, we use the open-source LLaVA-1.5. Specifically, we split our OEQ benchmark into a training set comprising 80% (120 VH instances for each VH mode) and a testing set comprising 20% (30 VH instances for each VH mode) of the data. To make the testing set performance as close as possible to the full OEQ benchmark dataset performance before fine-tuning, for every VH mode, we randomly divide the VH instances into 80%/20% split 100 times and select the split whose testing set accuracy is the closest to the accuracy on the full OEQ benchmark.
We follow LLaVA-1.5’s limited task-specific fine-tuning setting. However, we unfreeze the vision encoder since VH may result from its incorrect embedding vectors for images. We set the learning rate to 4e-6 during fine-tuning; and we fine-tune on the 960 training VH instances for one epoch (more hyperparameters and fine-tuning details are shown in Appendix F). Our fine-tuning took only 18 minutes on a single A6000.
Fine-tuning Results Table 6 shows the accuracy of LLaVA-1.5 and fine-tuned LLaVA-1.5 on the testing VH instances of our OEQ and YNQ benchmarks. Table 6 also shows the results on MME (Fu et al., 2023) and POPE (Li et al., 2023), two popular existing benchmarks (not necessarily VH) to evaluate the performance of an MLLM. MME evaluates the perception and cognition abilities of MLLMs. The scores for MME Perception and MME Cognition are the sum of scores across the corresponding subtasks, and the total scores are 2,000 for MME Perception and 800 for MME Cognition. The POPE score represents the average F1score on random, popular, and adversarial splits of POPE. We find that fine-tuned LLaVA-1.5 achieves higher average accuracy than LLaVA-1.5 in both our benchmarks, while they achieve comparable results on MME and POPE benchmarks. Our results indicate that fine-tuning an MLLM on our benchmark makes it less likely to hallucinate. We exhibit several cases as qualitative results, showing fine-tuning process mitigates visual hallucinations in Appendix B.3.
Ablation Study on Fine-tuning Figure 6 shows ablation study results on the MME benchmark and
the testing set of our YNQ benchmark when finetuning LLaVA-1.5. We use our YNQ benchmark instead of OEQ because it supports automatic evaluation. Since an MLLM has three key components: vision encoder, vision-language connector, and LLM, we explore fine-tuning different components. Our results show that fine-tuning all components achieves the best overall results. As for finetuning epochs, the results show that fine-tuning for one epoch minimizes overfitting to our benchmark, retaining the best performance on MME. For learning rate, both excessively large and small learning rates lead to a decline in performance on our benchmark and MME.

heading: 5 Related Work
text: Hallucinations Hallucinations are well-known issues for generative AI, including LLMs (Ji et al., 2023; Huang et al., 2023a), MLLMs (Liu et al., 2024b; Rawte et al., 2023; Tong et al., 2024), and text-to-image generative model (Tong et al., 2023).
In general, hallucination refers to a generative model imagines factually incorrect details in its response for a given input. VH occurs when an MLLM imagines incorrect details about an image in visual question answering.
VH Benchmarks in MLLMs Prior works have tried to benchmark MLLMs’ VHs (Li et al., 2023; Liu et al., 2024a; Fu et al., 2023; Tong et al., 2024). However, they collect VH images only from existing image datasets. This limits the diversity of VH images. Moreover, existing image datasets may have been used to pre-train an MLLM, leading to data contamination (Jacovi et al., 2023; Sainz et al., 2023). Our VHTest can generate a diverse set of new VH images that do not appear in existing benchmarks. Moreover, the shape and size VH modes are formulated by us for the first time.
Mitigating VH in MLLMs Existing works on mitigating VHs in MLLMs can be categorized into fine-tuning-phase and testing-phase mitigation. Fine-tuning-phase mitigation focuses on improving the fine-tuning data quality (Wang et al., 2024; Liu et al., 2024a) and/or model structure (Tong et al., 2024). These works typically freeze the vision encoder during fine-tuning, following the standard fine-tuning setting of LLaVA-1.5. We find that finetuning the vision encoder together reduces VHs in MLLMs. Testing-phase mitigation leverages prompt engineering with more visual evidence (Li et al., 2024) or correction tools for hallucinated responses (Yin et al., 2023). Testing-phase mitigation
is complementary to fine-tuning-phase mitigation.

heading: 6 Discussion
text: As shown in Table 2, Table 3, and Table 7, GPT4V, LLaVA-1.5, and MiniGPT-v2 all exhibit extremely low accuracy, with an average of 0.229 on our OEQ benchmark. Moreover, all seven mainstream MLLMs achieve accuracy levels close to random guessing on our YNQ benchmark, with even the state-of-the-art GPT-4V only reaching an accuracy of 0.611. Our benchmarks effectively reveal visual hallucinations within MLLMs and are more challenging than most existing benchmarks for MLLMs.
We attribute such high challenge level to the adversarial construction idea during VH instance generation, which is similar in spirit to adversarial examples (Szegedy et al., 2013) commonly used to test and improve model robustness. Unlike most existing MLLM benchmarks that are limited to the image space of existing image datasets like COCO, VHTest derives text descriptions from initial VH instances, which play a role of “adversarial direction”. Then VHTest exploits text-to-image generative models to generate challenging VH instances following such adversarial direction. This moves beyond the image space of limited existing image datasets and into the manifold of text-to-image generative models, thereby making it possible to generate more challenging VH instances. Thus, our VHTest offers a new perspective for constructing MLLM benchmarks for future research.

heading: 7 Conclusion
text: We propose VHTest to generate VH instances to test MLLMs. We collect VH benchmarks using VHTest and we find that state-of-the-art MLLMs exhibit high hallucination rates on our benchmarks. Moreover, fine-tuning MLLMs on our benchmark
reduces hallucination without sacrificing their other performance/capability.

heading: 8 Limitations and Future Work
text: We acknowledge that our VHTest still requires human workers to manually generate a questionanswer pair for an automatically generated VH image. An interesting future work is to make VHTest fully automatic so it can generate as many VH instances as needed.

heading: A Generated Text Descriptions of VH Modes
text: The following shows the text description for each VH mode generated by our VHTest.
1. Existence VH: In existence hallucination, a multi-modal large language model (MLLM) may encounter two types of hallucinations. The first type is when an object exists in an image, but the MLLM asserts that it does not exist in the picture, called negation existence hallucination. The other type is when a certain object does not exist in the image, but the MLLM creates or infers details, objects, or its attributes in an image, known as extrinsic hallucination.
As for the negation existence hallucination, an MLLM may exhibit challenges in recognizing the existence of certain objects within an image, particularly when these objects are not prominently featured within the picture or are partially obscured. The issue becomes apparent when the model disregards objects that are present but do not constitute the primary focus of the image. Factors contributing to negation existence hallucination include objects being small, distant, having low contrast with the background, or being located at the periphery of the frame. Such conditions can cause the MLLM to miss or ignore these elements, leading to an incomplete or inaccurate understanding of the scene. Negation existence hallucination is particularly evident in complex environments where multiple objects coexist but some are less dominant or visually prominent.
Extrinsic hallucination occurs when an MLLM creates or infers details, objects, or attributes in an image that are not actually present. This is often due to the model’s reliance on learned patterns and associations from its training data rather than the specific content of the image it’s analyzing. MLLMs may "hallucinate" details or objects based on what they have learned from other contexts, leading to extrinsic hallucinations. Also, the model readily associates certain scenes with typical objects even when those objects are not present, especially for scenes containing other complex patterns.
2. Shape VH: A multi-modal large language
model (MLLM) misconstrues shapes, particularly when typical simple shape and undulating strange shape are crowded. For example, a plate contains a banana and other fruit, some of which is undulating and coiled. When faced with non-standard shapes, it often simplifies them to more common, recognizable forms due to biases in training data. In instances where multiple shapes are present, especially with varying levels of detail or color intensity, the MLLM might get diverted towards the more attention-grabbing elements, overlooking or misreading other shapes in the process. Additionally, the MLLM faces difficulty in discerning subtle differences in shapes, often generalizing them into broad categories based on prominent features. These biases indicate a challenge in the MLLM’s ability to accurately interpret and distinguish between shapes, which could lead to incorrect generalizations and potential misinterpretations of image data.
3. Color VH: A multi-modal large language model (MLLM) struggles with recognizing or accurately identifying colors or color variations present in an image. It can misinterpret the sequence of color transition, omitting or inaccurately describing colors. This can be due to factors like subtle variations, intertwined colors, lighting, shadows, or adjacent color interference. Moreover, the recognition of color in little items may be more challenging, especially when the MLLM’s attention is attracted by a multitude of various elements inside the image.
4. Orientation VH: Orientation hallucination in a multi-modal large language model (MLLM) often results from a combination of factors that lead to misinterpretation of the direction in which objects, like vehicles, are facing. The primary issues include perspective ambiguities where 3D spatial orientation is not easily conveyed in a 2D image, dense object overlap, and environments with complex, busy backgrounds that distract from or mimic the shape and color of the target object. The MLLM might also falter when the object of interest is situated among others facing various directions, confusing the model’s directionality cues. Such errors could stem from the MLLM
within complex visual contexts.
5. OCR VH: OCR hallucination often stems from a complex interplay of factors. A multi-modal large language model (MLLM) falters when faced with unconventional visual scenarios, such as irregular character spacing, similar-looking letters, vertical arrangement of characters, or interference from nearby visual elements. These issues are compounded when the text contains intentional misspellings, typos, or uncommon character combinations that the system attempts to autocorrect based on standard language patterns. The ability to accurately recognize characters is further challenged by the influence of adjacent characters and the overall visual context.
6. Size VH: Size comparison in images requires the multi-modal large language model (MLLM) to recognize and accurately compare objects based on visual scale. Challenges in this task arise from factors like perspective, distortions, overlapping of objects, intricate patterns, complex backgrounds, discrepancies in expected real-world proportions, and excessive focus on foreground objects that causes the size of enormous background objects to be overlooked. Especially, when conducting a comparison among multiple objects of similar types and sizes, with other different types of objects in the scene that disturb the MLLM’s attention, the task becomes even more challenging. The MLLM misjudges which object is larger or might rank objects incorrectly in terms of size.
7. Position VH: A multi-modal large language model (MLLM) encounters difficulties in accurately assessing the spatial positioning relationship between objects within an image. This is exacerbated in scenarios where objects are placed in non-linear configurations, such as circular layouts, which can disrupt the model’s ability to apply standard left-toright reading patterns. Moreover, when the objects are set against complex or similar backgrounds that lack clear demarcation lines, the MLLM’s spatial parsing capabilities can be further compromised. Contributing factors such as overlapping objects, inconsistent scaling, and deceptive perspective or angled viewpoints also intensify this challenge. In
addition, the presence of shadows or uneven lighting may cast ambiguity on the object’s precise location, thereby leading to misinterpretation. Such spatial hallucination can be attributed not just to the inherent complexity of object arrangement but also to the MLLM’s processing of visual cues that inform depth, orientation, and the relationship of elements within the image space.
8. Counting VH: A multi-modal large language model (MLLM) has difficulty in accurately counting the quantity of specific elements or attributes in an image, especially when the objects are closely packed, overlap, are of different sizes, are partially visible, or have varying orientations. The task becomes increasingly complex when attempting to count specific subtype objects within a larger type category. Such conditions can lead the model to overlook certain objects, mistakenly merge similar items, or misinterpret the image data, thereby providing an inaccurate counting of numbers, amounts, or values.

heading: B Additional Results
text: B.1 Inter-annotator Agreement
Because annotating the whole benchmarks relies on heavy human labor, we randomly select 100 VH instances from our YNQ benchmark to measure the inter-annotator agreement. Specifically, four independent human annotators label these 100 VH instances. Based on these labeling results, we report Fleiss’ Kappa (κ) to indicate the inter-annotator agreement. We obtain κ = 0.958 (0.81 ≤ κ ≤1.00), showing a very high degree of agreement.
B.2 More MLLMs Evaluation on VHTest
We evaluate 4 additional MLLMs on the YNQ benchmark: the Gemini-pro-vision (Gemini) (March 2024), ShareGPT4V-13b (Chen et al., 2023b), InstructBLIP-13b (Dai et al., 2023), and Qwen-VL-Chat-7b (Bai et al., 2023), as shown in Table 7. Together with Table 3, GPT-4V shows the highest average accuracy of 0.611 among these 7 MLLMs.
B.3 Qualitative Results on Fine-tuning
To better elucidate the efficacy of fine-tuning for mitigating the visual hallucinations within MLLMs,
we report some responses from the LLaVA-1.5 before and after fine-tuning, as shown in Figure 8. In these cases, before fine-tuning, LLaVA-1.5 misrepresented the shapes of objects by assigning irrelevant shapes to them, i.e., asserting that the banana in Figure 8 is heart-shaped; LLaVA-1.5 was unable to precisely discern the relative size of objects, as demonstrated in Figure 8 where it wrongly pinpointed the second-largest chair to the right side of the image. Fortunately, the fine-tuned LLaVA-1.5 mitigates these flaws. As shown in Figure 8, LLaVA-1.5 can accurately describe the curved shape of the banana and correctly pinpoint the two largest chairs in the image. We observe such improvements in response quality across the entire testing set, indicating the effectiveness of our method.

heading: C Prompts Used in VHTest
text: We use the following prompt in Figure 9 for text description generation in Step II, based on a successful initial VH instance.
We utilize the following prompt in Figure 10 for text description generation in Step II, given an unsuccessful initial VH instance.
We use the following prompt in Figure 11 for text description integration in Step II.
We use the following prompt in Figure 12 for a text-to-image generative model to generate a VH image in Step III.
We use the following prompt in Figure 13 for an LLM to generate prompts for text-to-image generative models. We only append extra text at the end compared with the prompt in Figure 12.

heading: D Question Templates
text: We provide question templates for all eight VH modes below.
Question templates for existence VH mode
1. Could you describe [object] in the picture? 2. Describe the [property] of [object] in the picture.
Question templates for shape VH mode
1. Describe the shape(s) of [object] in the picture. 2. In the photo, is [object] depicted with a conventional, typical shape?
Question templates for color VH mode
1. Describe the color(s) of [object] in the picture. 2. Describe [n] main color(s) of [object] in the picture. 3. Describe all the colors of [object] in the picture. Do not miss any color. 4. Describe the original color of [object] in the picture.
Question templates for orientation VH mode
1. From the camera/viewer’s perspective, please choose the direction that is most proximate to the orientation in which [object] is facing in the picture. [Left, 45 degrees leftwards away from the camera/viewer, Away from the camera/viewer, 45 degrees rightwards away from the camera/viewer, Right, 45 degrees rightwards towards the camera/viewer, Towards the camera/viewer, 45 degrees leftwards towards the camera/viewer] 2. From the camera/viewer’s perspective, please choose the direction that is most proximate to the orientation in which [object] is facing in the picture. [Left, Left and upward within the pictorial plane, Upward within the pictorial plane, Right and upward within the pictorial plane, Right, Right and downward within the pictorial plane, Downward within the pictorial plane, Left and downward within the pictorial plane] 3. From the camera/viewer’s perspective, please choose the direction that is most proximate to the orientation in which [object] is facing in the picture. [Away from the camera/viewer, Towards the camera/viewer]
Question templates for OCR VH mode
1. Read the (English) character(s)/word(s)/ caption [at WHERE] in the picture.
Question templates for size VH mode
1. Could you identify and locate [object] with the largest/smallest size in the picture? 2. Could you identify and locate [object] with the tallest/shortest height in the picture? 3. Could you identify and locate [object] with the second largest/smallest size in the picture? 4. Could you identify and locate [object] with the largest/smallest and the second largest/smallest size in the picture? 5. Could you identify and locate [object] with the tallest/shortest and the second tallest/shortest height in the picture? 6. Could you identify and locate the 2/3/4 largest/smallest [object] in the picture? 7. Which one is larger/smaller/taller/shorter, [object A] or [object B]? 8. What is the largest [object] in the picture?
Question templates for position VH mode
1. Is [object A] to the left/right of/on the top of/being placed on/above/under/inside/outside [object B]? 2. Which one is on the left/right/top, [object A] or [object B]? 3. Which one is closer to/further from the camera or viewer perspective, [object A] or [object B]? 4. Which one is positioned under/positioned above/closer to [reference object], [object A] or [object B]? 5. Regardless of the positional relationship of the actual scene in reality, from the camera’s perspective in the photo, is [object A] positioned above/below [object B]? 6. Is [object A] on and touching [object B]?
Question templates for counting VH mode
1. How many [object] are depicted/there/visible/can be seen [at WHERE] in the picture?

heading: E Special Cases in Step III
text: In existence VH mode, a human worker uses the prompt in Figure 14 to find non-existent objects in a VH image with the aid of an MLLM, such as GPT-4V. The non-existent objects found in this way are more likely to trigger VHs in MLLMs.

heading: F Details of Fine-tuning Experiments
text: F.1 Hyperparamters When fine-tuning LLaVA-1.5 on the training set of VHTest dataset, we follow LLaVA-1.5’s limited task-specific fine-tuning setting. Additionally, we unfreeze the vision encoder. The hyperparameters are shown in Table 8.
F.2 Pre-processing of the Training Set As mentioned in (Liu et al., 2023), the short-form data overfit an MLLM behaviorally to short-form answers. We follow this work to do pre-processing on both counting and position VH modes. To be more specific, for counting VH mode instances, we use the prompt below in Figure 15 for an LLM (e.g., GPT-4) to modify every number reference answer into a reference sentence. As for position VH mode instances, we add the sentence “Answer the question using a single word or phrase.” after all the questions in every position VH instance.

heading: Future Work
text: An interesting future work is to make VHTest fully automatic so it can generate as many VH instances as needed