Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024)

Abstract: Though advanced in understanding visual information with human languages, Large VisionLanguage Models (LVLMs) still suffer from multimodal hallucinations. A natural concern is that during multimodal interaction, the generated hallucinations could influence the LVLMs’ subsequent generation. Thus, we raise a question: When presented with a query relevant to the previously generated hallucination, will LVLMs be misled and respond incorrectly, even though the ground visual information exists? To answer this, we propose a framework called MMHalSnowball to evaluate LVLMs’ behaviors when encountering generated hallucinations, where LVLMs are required to answer specific visual questions within a curated hallucinatory conversation. Crucially, our experiment shows that the performance of open-source LVLMs drops by at least 31%, indicating that LVLMs are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions. We term this phenomenon Multimodal Hallucination Snowballing. To mitigate this, we further propose a training-free method called Residual Visual Decoding, where we revise the output distribution of LVLMs with the one derived from the residual visual input, providing models with direct access to the visual information. Experiments show that our method can mitigate more than 24% of the snowballed multimodal hallucination while maintaining capabilities.1

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11991–12011 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Large Vision-Language Models (LVLMs) have shown remarkable abilities in observing and understanding the real world in human languages (Achiam et al., 2023; Zhu et al., 2023; Liu et al., 2023d; Ye et al., 2023b; Dai et al.). However, multimodal hallucinations, in which LVLMs provide
*Corresponding Author 1Resources will be available at https://github.com/
whongzhong/MMHalSnowball
advice independently. Green and red colors highlight the correct answer and hallucinations, respectively.
responses misaligned with the corresponding visual information, remain to be the Achilles’ heel (Cui et al., 2023; Kamath et al., 2023; Li et al., 2023b; Liu et al., 2023a; Lu et al., 2023; Rawte et al., 2023; West et al., 2023; Huang et al., 2023).
Previous research has revealed that hallucinations generated by large language models may accumulate due to models’ over-commitment to early mistakes, leading to more mistakes that they otherwise would not make (Zhang et al., 2023a; Azaria and Mitchell, 2023; Kang et al., 2023), especially for the user-model interaction scenarios such as conversation (Huang et al., 2022; Tian et al., 2024; Gong et al., 2023). However, the extent to which accumulated multimodal hallucinations mislead LVLMs into generating false claims requires further exploration. In this work, we conducted an investigation into this issue for the first time. As shown in Figure 1, we seek the answer to the question: When presented with a query relevant to the
11991
previously generated hallucination that contradicts the visual information, can models make the correct judgment when they could have given a correct answer independently? We conduct a preliminary study on GPT-4V (Achiam et al., 2023), LLaVA 1.5 (Liu et al., 2023c), and mPLUG-Owl2 (Ye et al., 2023b). Similar to the setting of Figure 1, given an image, we start a conversation by asking the model to describe the image in detail. When observing hallucinations in the LVLM’s responses, we continue to ask a relevant question according to the model-generated hallucination. In addition, we ask the same question separately to see if the model can answer it correctly without distractions. As demonstrated in Figure 2(a), we find that when the text context contains relevant hallucination, the model performance declines significantly, compared to the model response when asking the same question separately. We further select those question samples that the LVLM can correctly answer separately, and manually identify the response change when asking the same question with the related model-generated hallucinatory context. As Figure 2(b) depicts, we find that more than 59% of the answers are semantically the same as the generated hallucination, indicating that they were misled by the previously generated hallucinations.
To systematically investigate this phenomenon, we propose to identify whether the LVLM is misled by hallucinations via checking if a specific claim is flipped due to previous hallucinations. We design a framework called MMHalSnowball to construct hallucinatory visual conversations, where models
are required to answer the question based on the image and the hallucinatory conversation. The result shows that LVLMs’ multimodal hallucinations are easy to mislead the later generation because their strong language capabilities make them prone to be over-confident in the hallucinated context, thereby generating false claims that they normally would not support, which we term as Multimodal Hallucination Snowballing.
In addition to mitigating this issue, we further proposed a training-free decoding method called Residual Visual Decoding (RVD). By residual connecting the visual information and the current user instruction, distributions that emphasizing the visual information are derived to revise the original output distribution. Our RVD achieves more than 24% of improvements in reducing the multimodal hallucination snowballing while maintaining the contextual modeling ability.
2 Evaluating the Multimodal Hallucination Snowball Phenomenon
In this section, we design a question-answer task in the conversation scenario, where a model is first asked to describe a picture in detail and then answers a visual question. As shown in Figure 3, we propose the MMHalSnowball framework to carefully simulate hallucinatory conversations and evaluate whether the model generates a wrong answer due to the hallucinatory context. Next, we will describe our evaluation framework in detail, including conversation creation, experimental settings, and evaluation metrics. We experimentally analyze the multimodal hallucinations snowball in §2.7. The prompts used are listed in Appendix A.2.
2.1 Dataset Source We use the validation set of the GQA dataset (Hudson and Manning, 2019) as our data source, which contains a balanced aspect of visual questions that focuses on objective perceptional questions. We adopt images, question-answer pairs, and regional description annotations from the Visual Genome (Krishna et al., 2017). Note that we use its balanced validation set to minimize the impact of dataset contamination and language prior.
2.2 Hallucination Allocation To be more practical, we construct hallucinations based on the common types generated by LVLMs. Inspired by Wang et al. (2023a); Zhai et al. (2023), we categorize the hallucinations as follows:
whether they suffer from multimodal hallucination snowballing. Green and red color highlight the correct answer and hallucinations curated out of it, respectively.
• Existence Hallucination, which refers to the incorrect recognition of visible objects in the image or the belief that specific visible objects are absent in the image. • Attribute Hallucination, which refers to the inaccurate characterization of objects and misrepresentations of attributes such as color, shape, size, and actions. • Relation Hallucination, which refers to the inaccurate depiction of the relationships or interactions among objects, including erroneous interaction states, relative positions, and spatial positions of objects relative to the image. • Imagination Hallucination, which refers to the erroneous imagination of objects in the picture that do not appear.
To incorporate hallucinations, we first utilize ChatGPT (OpenAI, 2022) to rewrite a fact sentence
that best describes the question-answer pair. In addition, the annotated regional descriptions and the fact sentence are used to generate an image description. The ChatGPT is prompted to ensure the image description semantically entails the fact sentence. Hallucination can be created by properly modifying the fact sentence. Our goal is to make the answer to the original question no longer correct according to the modified fact sentence. However, not all types of hallucination will make the original answer invalid (e.g. modify the fact sentence "The color of the trousers is blue" to "the color of the bike is blue" introduces an imagination hallucination, but won’t invalidate the answer to the question: "What color are the trousers that this boy is wearing in the image?"). To match the hallucination errors in the curated contexts with the corresponding question-answer pairs, We then
allocate a proper hallucination type from the above definition to each fact sentence. Appendix A.1 shows details about the rules of allocating proper hallucination types.
2.3 Hallucination Creation In this part, we describe how we utilize the question-answer pair, the fact sentence, and the regional descriptions to generate hallucinatory image descriptions. Rather than directly modifying the fact sentence according to the hallucination type to create hallucinations, we find it more stable to ask the ChatGPT to rewrite a hallucinatory answer that contradicts the original answer. Then, the fact sentence, as well as all the regional descriptions are heuristically modified to hallucinatory ones according to the hallucinatory answer. With the hallucinatory fact sentence and hallucinatory regional descriptions as inputs, the ChatGPT is asked to generate a detailed image description that entails the hallucinatory fact. the original answers Y + = {y+1 , y+1 , ..., y+n } and the rewritten hallucinatory answers Y − = {y−1 , y−1 , ..., y−n } are kept for evaluation, where n represents the dataset size.
2.4 Conversation Construction Before constructing a hallucinatory conversation, we should ensure that the generated hallucinatory answer and descriptions contradict the image content, while the hallucinatory description supports the hallucinatory answer. To do this, we provide ChatGPT with descriptions, answers, and their corresponding hallucinatory ones to check if the modification and generation meet our requirements. We also check if the image description generated in Section 2.2 entails the fact sentence. See Figure 12 for the prompt used. Note that only those descriptions that conflict with the original answer but can deduce the hallucinatory answer will be kept.
After checking, we utilize the generated hallucinatory descriptions and the question-answer pairs to construct a question-answering conversation, as Figure 3 step 3 shows. Conversation examples for each hallucination type are in Appendix A.5.
2.5 Statistics
With our meticulous data curation and checking process, Our curated dataset D contains 4,973 samples in total. The detailed sample number for each hallucination type is as Table 4 shows. As Figure 5 shows, the diverse nature of the GQA dataset is maintained.
To check the effectiveness of the modifications made by our framework, as Figure 5 left shows, we sample 400 data and manually review them by several professionals. Our generated hallucinatory answers and conversations mostly meet our expectations, Please refer to Appendix A.3 for more details about manual checking.
2.6 Evaluation
To gain a deep understanding of the LVLMs’ multimodal hallucination snowballing, given visual question-answering pairs from our dataset, We generate model responses under two different settings as Figure 1 shows and compare the results under these two conversation settings. The first setting is that the model generates the response to the question in our curated corresponding hallucinatory conversation, which we refer to as HalluConv.. The second is that the model answers the same visual question alone, without the distraction of hallucinatory context, which we refer to as CleanConv.. Since LVLMs’ response format can be diverse due to the ambiguous query prompt, it might make the automatic evaluation result slightly imprecise. To address this, we follow (Liu et al., 2023c) to add a formatting prompt right after the question: "Please answer the question using a single word or phrase.", namely Formatting Prompt setting. The user input with the question only is named as Question Prompt. Note that we conduct experiments with Formatting Prompt if not specified.
2.6.1 Evaluation Metrics In this part, we introduce our evaluation metrics. First, to evaluate the correctness of each generated answer, we adopt the following criteria: Entailment Matching Score: Considering both the original answer and the hallucinatory answer were short, while models tends to generate longer
CleanConv. and HalluConv settings. Numbers that are highlighted orange represent the model performance drop caused by hallucinatory conversation, compared to the model performance under CleanConv. setting. The results in bold and underlined represent the best and the second-best results, respectively. All experiments are implemented under a zero-shot setting to avoid the bias introduced by demonstrations.
answers with explanations. We evaluate the correctness for the ith sample by checking if the answer is entailed in the generated response:
Scorei = 1 if yi in ŷi else 0, (1)
where yi and ŷi stand for the expected answer and the generated response, respectively. With a proper scoring method for one sample, we can calculate the overall accuracy with the following method: Accuracy (Acc):
Acc(Y, Ŷ ) = ∑n
i=1 Scorei(yi, ŷi) n , (2)
where Acc(Y, Ŷ ) represents the model’s accuracy score over the entire dataset. Flip Rate (FR):
In order to systematically measure whether one model is affected by the hallucination snowballing phenomenon, we propose the FR to evaluate how many model responses are misled by hallucinatory context and are matched with our curated hallucinatory answers:
FR =
∑ i∈D+ Scorei(y − i , ŷ − i )
Acc(Y +, Ŷ ) , (3)
D+ = {i|Score(y+i , ŷ+i ) = 1, ŷi ∈ Ŷ , y+i ∈ Y +}, (4) where Ŷ + = {ŷ+1 , ŷ+1 , ..., ŷ+N} and Ŷ − = {ŷ−1 , ŷ−1 , ..., ŷ−N} represent generated answers under CleanConv. and HalluConv. settings, D+ represents the sample indexes that the LVLM correctly answers in the CleanConv. setting.
Furthermore, we designed a more generalized flip-rate metric named weak flip-rate(WFR) which only evaluates how many model responses are distracted by hallucinatory context and conflict with the original answers:
WFR =
∑ i∈D+(1− Scorei(y+i , ŷ−i ))
Acc(Y +, Ŷ +) , (5)
2.6.2 Models We investigate the multimodal snowballing phenomenon in the following mainstream LVLMs: LLaVA-1.5 (Liu et al., 2023c), MiniGPT-4 (Zhu et al., 2023), MiniGPT-v2 (Chen et al., 2023a), InternLM-XComposer (Zhang et al., 2023b), ShareGPT4V (Chen et al., 2023b), CogVLM (Wang et al., 2023b), mPlug-Owl (Ye et al., 2023a),
mPlug-Owl2 (Ye et al., 2023c), Qwen-VL-Chat (Bai et al., 2023), Otter (Li et al., 2023a), IDEFICS (Laurençon and Strien, 2023), InstructBLIP (Dai et al.) and GPT-4V (gpt-4-vision-preview)(Achiam et al., 2023). All experiments are completed under a zero-shot setting. Please refer to Appendix A.4 for more generation details.
2.7 Do LVLMs Suffer from Multimodal Hallucination Snowballing?
To answer this question, we compare the model responses under the conversation settings of HalluConv. and CleanConv., as Section 2.6 describes. The results are depicted in the Table 1. Though advanced in answering visual questions even in a zeroshot manner (See accuracy in CleanConv.), most models struggle to stick to their judgment when there are specious hallucinations in the context (See accuracy in HalluConv.), resulting in extremely low accuracy. For LLaVA-1.5, ShareGPT4V, mPlugOwl2, and InstructBLIP, despite their advanced model ability, they still suffer an over 50% performance drop. However, we also recognize that GPT4V is significantly less affected by hallucinations. We observed a correction process in the responses of GPT-4 (See Appendix B.2 for examples), indicating that it is capable of paying attention to visual information to a certain extent and realizing that some hallucinations have been generated in the conversation. In addition, we find that GPT-4 often refuses to answer the user question due to its strict safety protocol, especially in the Clean Conv. setting (around 12%), indicating a potential cause of such a comparably low accuracy. But in general, all the LVLMs suffer from multimodal hallucination snowballing at different levels. What’s more, a high flip rate indicates that the model responses are easily misled by the hallucinatory conversation, even when the model can make a correct claim in CleanConv. setting. An even higher weak flip rate is observed, which shows that LVLMs’ responses are corrupted due to the hallucinatory context. Hence, comparing the same LVLMs with different scale LLM backbones, we find no significant performance improvement in mitigating the multimodal hallucination snowballing, except for the InstructBLIP.
Comparing the experiments between two different query prompts, we find that the Formatting Prompt shows clearer instructions, which not only improves question-answering ability but also eases the multimodal hallucination snowballing
phenomenon for most of the LVLMs. We further present the accuracy of two different conversation settings and the flip rate for each hallucination type in figure 5. The result shows that existence, attribute, and imagination hallucinations are easier to snowball. We even observe a nearly 100% flip rate on the imagination hallucination where LVLMs readily accept objects that are mistakenly imagined to exist, which could attributed to the LVLMs’ nature to generate positive response (Liu et al., 2023b). while the relation hallucinations have a higher probability of being correct while answering the question. For detailed results, please refer to Appendix B.1.
2.8 Will LVLMs Be Affected by the Hallucination-Free Context?
Compared to CleanConv. setting, where the conversation context only contains one image and one user question, LVLMs under HalluConv. setting are required to answer the same user question with an additional round of conversation. How does a longer context length affect the model performance? To answer this question, we further create two conversation settings that have similar context length to HalluConv. setting, in which there is also an additional conversation round but without hallucinatory content related to the user question. Specifically, we first replace the hallucinatory descriptions in Halluconv. setting with the image descriptions generated in Section 2.2, which are semantically consistent with the fact sentence. We

heading: Model CleanConv.↑ FactConv.↑ IrrConv.↑ HalluConv.↑
text: tings. orange and green numbers represent the model performance drop and improvement in different conversation settings, compared to the model performance under CleanConv. setting. The results in bold and underlined represent the best and the second-best results, respectively.
name the resulting new conversation setting as FactConv. setting. In addition, we replace the 1st round conversation in HalluConv. with a single questionanswer pair that is irrelevant to any specific visual information in the image, namely IrrConv. setting (See Appendix B.3 for more details). The results are as Table 2 shows. From the results, we can observe that all the models benefit a lot from a correct image description, which further proves that LVLMs tend to rely on text context when there is text format visual information that can help to generate the response. Such nature could potentially lead to the hallucination snowballing with a hallucinatory conversation. What’s more, when the context provides no useful information, the models’ abilities are not severely influenced by the context, which further indicates the performance drop in HalluConv. setting is caused by hallucination snowballing, not the context length.
3 Residual Visual Decoding
From the phenomenon of multimodal hallucination snowballing, we find that LVLMs tend to condition on text context when there are plausible clues to help make responses, thereby ignoring the visual information and could be easily misled by erroneous context. To remedy this, we manage to emphasize the visual information during the inference process without additional training or external tools under the multi-turn conversation scenario.
3.1 Residual Visual Predictions Given a visual input v, a dialog history h, and the current text query x, one LVLM parametrized by θ generates a response y token-wisely. With generated tokens y<t up to time step t − 1, the output distribution in time step t is formulated as pθ(yt|v, h, x, y<t), where the output token yt is sampled from the output distributions:
yt ∼ pθ(yt|v, h, x, y<t) = softmax(logitθ(yt|v, h, x, y<t)),
(6)
Since the hallucinatory context could interfere with the process of reasoning over the visual input, we first construct an input that residual connects the visual input v with the current text query x, and derive an output distribution from it:
pθ(yt|v, x, y<t) = softmax(logitθ(yt|v, x, y<t)), (7)
in which the output distribution will naturally shift from dependence on text context to reliance on visual information. We term it the Residual Visual Predictions, which are based entirely on visual information and the query while sacrificing attention to the text context.
3.2 Residual Visual Decoding In order to put an emphasis on the visual information under a multi-turn visual text conversation scenario, inspired by (Leng et al., 2023; Liu et al., 2021), we introduce Residual Visual Decoding (RVD), where residual visual predictions are utilized to enhance the perception of the visual information. The revised distribution pRVD is formulated as:
pRVD(y|v, h, x) = softmax(αlogitθ(y|v, x) +(1− α)logitθ(y|v, h, x)), (8)
where a larger α indicates a higher model focus on the visual information. Note that when the length of dialog history h is 0, the RVD degenerates to the regular decoding.
3.3 Adaptive Distribution Blending However, as we tune up the α, the text context gets to be ignored when generating responses, which possibly does harm to the model’s inherited contextual ability. To preserve the contextual ability while tackling the hallucination snowballing, we propose to adaptively adjust the scaling parameter. Specifically, we derive an output distribution
pθ(y|x) given the current user query x only, and calculate the Jensen-Shannon divergence (JSD) between it and residual visual predictions, which evaluates the similarity between two output distributions:
τ = JSD(pθ(y|v, x)||pθ(y|x)), τ ∈ [0, 1], (9)
where τ is the JSD score between pθ(y|v, x) and pθ(y|x). We suspect that when responding to the query depends on the visual information v, τ gets larger, since the latter is barely making guesses. Meanwhile, when responding to the query depends on the dialog history h, the corresponding two distributions tend to make guesses. However, they still have access to the nearest user query from the current round of conversation. Thus, We assume that conditioned on these two output distributions tend to make similar guesses so that the τ will get smaller. Therefore, we dynamically adjust the α with τ and a scaling factor β:
α = Min(β ∗ τ, 1), (10)
With the dynamic adjusted α, we can adaptively blend the residual visual distribution into the original output distribution with equation (8).
3.4 Experiments
By blending the residual visual distribution into the original output distribution, the models’ contextual ability could be harmed. Inspired by Chen et al. (2023c), to quantitatively evaluate the LVLMs’ contextual ability with our pipeline, we construct a multiple choice task called Who Provide This Image (WPI). Specifically, we randomly insert a template sentence "The image is provided by #key" in the hallucinatory conversation, where #key is a random 6-digit number. We then change the corresponding question to "Who provides this image?". An LVLM that can correctly access the context will have over 90% accuracy in answering this question. For more details, please refer to Appendix A.6.
As a result, We test our proposed RVD in our proposed multimodal hallucination snowballing evaluation and the aforementioned WPI task to evaluate its ability to alleviate the multimodal hallucination snowballing while maintaining contextual ability.
3.4.1 Baselines To show the effectiveness of our proposed RVD, we compare our method with the following strategies:
orange and green represent the model performance drop and improvement, respectively. The results in bold represent the best results, respectively.
• Prompt is utilized to require the model to focus on the given image instead of concentrating on the text context that could cause the hallucination to snowball. Specifically, we explicitly ask the model with the following query: {#Question, Please answer the question based on the given image.}.
• Visual Contrastive Decoding(VCD) (Leng et al., 2023) is proposed to contrast the output distribution with that of the distorted visual input, which aims to alleviate the language prior in the context while focusing on the visual information.
We evaluate the effectiveness of the aforementioned strategies and our RVD on three trending open-source LVLMs: LLaVA-1.5, mPlug-owl2, and ShareGPT4V. We set the β = 2 if not specified.
3.4.2 Experiment Results
The results are shown in Table 3. We find that incorporating the prompt methods will do harm to the model performance, which might be because of the inability of LVLMs to follow complex instructions. Though shown to be effective in correcting the snowballed hallucination, the VCD contrasts the output distribution with the distorted visual input, which could do harm to the model performance when the context is utilized to respond to the query. However, by dynamically emphasizing the visual information whenever needed, our proposed RVD makes a large accuracy improvement in overcoming the multimodal hallucination snowballing while maintaining contextual ability.
3.4.3 Effect of Parameters We evaluate the effect of our proposed hyperparameters α and β. The results are shown in Figure 6 and 7. First, we remove the adaptive distribution blending and adjust the α manually, the result shows that a larger α clearly revises the output distribution more towards the golden visual information. However, the context is ignored in return. With adaptive distribution blending, the model performance is more balanced when we enlarge the β, which won’t cause a large performance drop on contextual abilities. See Appendix B.4 for more experiment results.
4 Related work
4.1 Large Vision-Language Models
Inspired by the recent success of large language models (LLMs) (Zhao et al., 2023), researchers have devoted significant effort to integrating LLMs into vison-language models to utilize their powerful language understanding and generation capabilities (Wu et al., 2023). In addition to the advanced capabilities demonstrated by closed-source models such as GPT-4V(Achiam et al., 2023), open-source large vision-language models(LVLMs), building upon powerful open-source LLMs such as LLaMa (Touvron et al., 2023) and Vicuna (Chiang et al., 2023), have adopted a powerful instruction following abilities to tackle visual-language tasks in a zero-shot manner (Zhu et al., 2023; Liu et al., 2023d; Dai et al.; Ye et al., 2023b). Possessing both visual perception abilities and language capabilities, LVLMs are further utilized to perform real-world tasks,
such as tool-using (Liu et al., 2023e), web browsing (Zheng et al., 2024), and autonomous driving (Xu et al., 2023). However, current LVLMs still suffer from severe multi-modal hallucination problems (Liu et al., 2024), which brings challenges to evaluating and maintaining the reliability of LVLMs.
4.2 Multimodal Hallucination
Multimodal hallucinations (Liu et al., 2024) refer to the responses generated by LVLMs that are misaligned with the corresponding visual information. Multimodal hallucination can arise due to overfitting to specific patterns in the training data, inferior abilities to recognize the visual elements, or an inability to model the multimodal input. Li et al. (2023b), Lovenia et al. (2023), take the first step towards evaluating the hallucinations in the LVLMs. Furthermore, Liu et al. (2023b), Zong et al. (2023) and Liu et al. (2023a) show that LVLMs can be easily fooled and experience a severe performance drop due to their over-reliance on the strong language prior. In addition, efforts have been made towards mitigating multi-modal hallucinations by further finetuning or post-hoc rectify(Gunjal et al., 2023; Lu et al., 2023; Liu et al., 2023b; Zhou et al., 2023; Yin et al., 2023). However, current methods are unable to completely eliminate the hallucinations generated by models, yet no one has explored the subsequent impacts of the generated hallucinations. In this paper, we take the first step towards it by systematically evaluating the multimodal hallucination snowballing phenomenon and propose a training-free method to ease LVLMs from it.
5 Conclusion
In this paper, we raise the question of Whether LVLMs suffer from multimodal hallucination snowballing. We meticulously designed the MMHalSnowball framework to simulate hallucinatory conversations and study models’ behaviors when encountering hallucinations. Our investigation proved that LVLMs are being severely affected by hallucinations in the context, thus generating snowballed hallucinations. Further, we proposed the Residual Visual Decoding to alleviate the multimodal hallucination snowballing while maintaining its contextual abilities. However, our methods still have limitations when deployed to a generalpurpose assistant, which we left as future works.
6 Limitations
In this work, with a carefully designed evaluation framework, we have revealed that current LVLMs severely suffer from multimodal hallucination snowballing. We further proposed the RVD to mitigate the phenomenon. However, our work still has limitations. Firstly, despite the greater variety of hallucination snowballing phenomena in the realworld setting, the scenarios we focus on are still relatively simplistic. This is because constructing rich and diverse scenarios would be more difficult and would require a significant amount of effort. Secondly, instead of meticulously finding real hallucinations generated by each LVLM and constructing relevant question-answer pairs, we choose to conduct experiments on our simulated hallucinatory conversations. This is because the evaluation processes based on responses from a single LVLM will make it difficult to scale up the evaluation data and adapt to more models. Thirdly, our experiments are conducted on models of 7B and 13B sizes, and we evaluate our proposed RVD only on a few selected models. This is due to computational limitations. Fourthly, our proposed RVD is currently still limited in several conversation scenarios. We will further explore expanding this method to more diverse conversation scenarios.
7 Acknowledgments
Xiaocheng Feng is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was supported by the National Key R&D Program of China via grant No. 2021ZD0112905, National Natural Science Foundation of China (NSFC) via grant (62276078, U22B2059), the Key R&D Program of Heilongjiang via grant 2022ZX01A32, the International Cooperation Project of PCL, PCL2022D01 and the Fundamental Research Funds for the Central Universities via grant No. HIT.OCEF.2023018.
References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile visionlanguage model for understanding, localization, text reading, and beyond.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023a. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023b. Sharegpt4v: Improving large multimodal models with better captions. arXiv preprint arXiv:2311.12793.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023c. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An opensource chatbot impressing gpt-4 with 90%* chatgpt quality.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic analysis of hallucination in gpt-4v(ision): Bias and interference challenges.
W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards generalpurpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.06500.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790.
Anisha Gunjal, Jihan Yin, and Erhan Bas. 2023. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308.06394.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR).
Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s "up" with vision-language models? investigating their struggle with spatial reasoning.
Haoqiang Kang, Juntong Ni, and Huaxiu Yao. 2023. Ever: Mitigating hallucination in large language models through real-time verification and rectification. arXiv preprint arXiv:2311.09114.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73.
Hugo Laurençon and Daniel van Strien. 2023. Introducing idefics: An open reproduction of state-of-the-art visual langage model.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. Mitigating object hallucinations in large visionlanguage models through visual contrastive decoding. arXiv preprint arXiv:2311.16922.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03023.
Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023a. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v(ision), llava-1.5, and other multi-modality models.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023b. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565, 1(2):9.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023c. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023d. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023e. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437.
Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv preprint arXiv:2310.05338.
Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. 2023. Evaluation and mitigation of agnosia in multimodal large language models.
OpenAI. 2022. Introducing chatgpt.
Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models.
Yunjie Tian, Tianren Ma, Lingxi Xie, Jihao Qiu, Xi Tang, Yuan Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. 2024. Chatterbox: Multi-round multimodal referring and grounding. arXiv preprint arXiv:2401.13307.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023a. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023b. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079.
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2023. The generative ai paradox: "what it can create, it may not understand".
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth KY Wong, Zhenguo Li, and Hengshuang Zhao. 2023. Drivegpt4: Interpretable end-toend autonomous driving via large language model. arXiv preprint arXiv:2310.01412.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023a. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023b. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023c. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045.
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. 2023. Halle-switch: Controlling object hallucination in large vision language models. arXiv e-prints, pages arXiv–2310.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023a. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534.
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023b. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. 2023. Fool your (vision and) language model with embarrassingly simple permutations. ArXiv preprint, abs/2310.01651.
A Additional Experimental Details
A.1 Hallucination Allocation After carefully analyzing the question-answer pairs in the dataset, we manage to create an answer vocabulary for answers suitable for introducing relation errors. What’s more, we utilize Part-ofSpeech2 of the answer in the fact sentence to choose proper hallucination types. Specifically, we allocate attribute hallucination for those answers tagged as adjectives and verbs and allocate existence hallucination for those answers tagged as nouns. For imagination hallucination, instead of using the annotated question-answer pair, we provide ChatGPT with all annotated objects and ask ChatGPT to generate an object that is not present in the image but is reasonable to be in the corresponding scene. Then, we directly construct a question-answer pair with the template: "question: Is there a in the image? answer: No".
A.2 Prompts In this section, we list all prompts used during the process of constructing hallucinatory conversations, which include fact generation (Figure 9), conflict creation (Figure 10), description generation (Figure 11) and conflict verification (Figure 12). Note that we reuse the description generation prompt (Figure 11) to generate the ground image description by giving the annotated regional description and fact sentence.
A.3 Manual Checking We randomly select 100 data for each hallucination type in our curated dataset, 400 in total. We ask three annotators to check each of them from three aspects, as Table 5 depicts. The annotation results show that the generated hallucinatory description mostly meets our requirements.
A.4 Generation Details Through all our experiments, we follow a consistent generation configuration to ensure fairness. Specifically, we set the inference hyperparameter as follows: do_sample=True, temperature=1.0, top_p=0.95, top_k=None and num_beams=1.
A.5 Hallucinatory Conversation Example We list four examples to demonstrate curated hallucinatory conversation of each hallucination type, namely existence(Figure 13), attribute(Figure 14),
2We use Spacy to do the Part-of-Speech tagging.
relation(Figure 15), and imagination(Figure 16), where the ground answers are highlighted green and the hallucinated answers are highlighted red .

heading: A.6 Details of Who Provides This Image Task
text: We construct the Who Provides This Image (WPI) task to evaluate the contextual capabilities of LVLMs. To achieve this goal, we adopt a multichoice approach and judge the match by checking if the answer contains only the correct option (instead of the content). Both the correct option and interference option are randomly generated six-digit numbers, and the third option is "None of the options are correct". To further ensure fairness and effectiveness, the order of choices is also random. An example is shown in Figure 8.
B Additional Experimental Results
B.1 More Evaluation Results We show our detailed evaluation results for each hallucination type in Table 6.
B.2 GPT-4 Answer Examples We present GPT-4 answer examples with the Question Prompt. The first example is represented in Figure 17, which illustrates that GPT-4V is able to adaptively focus on golden visual information, and further identify and clarify the hallucinations in the previous hallucinatory description in some cases.
The second example is represented in Figure 18, which demonstrates that GPT-4V tends to refuse to answer some categories of questions, leading to difficulty in the evaluation and the degradation of the evaluation results.
B.3 Hallucination-free Context Experiment Details
In order to exclude the interference of irrelevant factors and to check whether LVLMs are affected by the hallucination-free context, we further set up two conversation settings, namely FactConv. and IrrConv. Corresponding examples are shown in Figure 19. We follow equation 3 and equation 4 to calculate FR and WFR metric, respectively, but we modify the definition of Ŷ − to represent generated answers under FactConv. or IrrConv. setting. We further present the full experiment result for these two conversation settings in Table 7
B.4 Effect of Different Similarity Measurement Methods
In our RVD, we choose JSD to evaluate the output distribution similarity, because it’s a symmetric metric that measures the difference between two distributions, with a range [0, 1], which fits our goal of adjusting the α (range is also [0, 1]) dynamically using the difference between two distributions. We also try to use the Kullback–Leibler
divergence (KLD) as the similarity measurement method, which is not a symmetric metric with a range [0,+∞] and can’t be directly applied to our Residual Visual Decoding (RVD). Specifically, to transform the range into [0, 1], We modify the Equation 9 to the following form:
τ = 1− EXP(−KLD(pθ(y|v, x)||pθ(y|x))). (11) We compare the experiment results of our RVD using KLD and JSD as the similarity measurement method on our proposed MMHalSnowball framework with LLaVA-1.5 7B. The results are as the Table 4 shows. We can observe from the table that RVD with KLD aggressively puts more emphasis on visual information, resulting in a better result in hallucination snowballing with smaller β, but a worse contextual ability. The result indicates that the JSD has a generally smaller value and is more balanced compared to the KLD in alleviating snowballed hallucinations while maintaining contextual ability.
B.5 Case Study In this part, we present some cases of the LLaVA1.5-7B model equipped with our proposed RVD. All cases are using the Question Prompt. We provide some case studies in Figure 20, one for each hallucination type, to demonstrate the effectiveness of our methods, where our RVD with LLaVA-1.57B successfully mitigated the snowballed hallucinations. In these examples, we can observe that with our proposed RVD, the model can focus more on the visual information to avoid generating snowballed hallucinations, rather than solely rely on the previously generated hallucinatory text and thus generate snowballed hallucinations. What’s more, in the example of Imagination Hallucination, the model with our RVD can even correct its previous mistakes, further illustrating the model’s contextual capability is preserved while avoiding hallucination snowballing.
M od
el
Im ag
in at
io n
E xi
st en
ce A
tt ri
bu te
R el
at io
n
C le
an C
on v.
H al
lu C
on v.
C le
an C
on v.
H al
lu C
on v.
C le
an C
on v.
H al
lu C
on v.
C le
an C
on v.
H al
lu C
on v.
A cc ↑
A cc ↑
FR ↓
W FR
↓ A
cc ↑
A cc ↑
FR ↓
W FR
↓ A
cc ↑
A cc ↑
FR ↓
W FR
↓ A
cc ↑
A cc ↑
FR ↓
W FR ↓ 7B LL M L L aV A -1 .5 81 .6 5 1. 14 ↓ 8 0. 51 98 .7 9 98 .7 9 60 .8 2 9. 22 ↓ 51 .6 0 79 .3 0 88 .0 5 55 .9 6 10 .1 8 ↓ 45 .7 8 78 .8 5 85 .2 1 83 .7 6 38 .0 9 ↓ 4 5. 6 7 57 .0 7 57 .6 1 M in iG PT -4 5. 69 0. 83 ↓ 4 .8 6 96 .0 0 96 .0 0 55 .0 5 7. 62 ↓ 47 .4 3 78 .5 8 88 .2 4 39 .0 7 2. 98 ↓ 36 .0 9 86 .6 5 94 .2 8 51 .4 4 11 .6 1 ↓ 3 9. 8 3 86 .2 8 87 .0 2 M in iG PT -v 2 67 .4 0 4. 70 ↓ 62 .7 0 94 .2 6 94 .2 6 52 .4 8 14 .4 5 ↓ 3 8. 0 3 64 .0 2 78 .8 9 47 .3 5 15 .5 6 ↓ 31 .7 9 59 .4 4 75 .7 0 78 .6 0 49 .3 9 ↓ 2 9. 2 1 46 .8 1 47 .1 0 In te rn L M -X C om po se r 49 .0 5 1. 06 ↓ 47 .9 9 98 .4 5 98 .6 1 46 .4 5 6. 21 ↓ 40 .2 4 79 .9 6 90 .6 5 35 .1 8 4. 06 ↓ 31 .1 2 83 .2 9 92 .0 0 43 .1 0 11 .9 1 ↓ 3 1. 1 9 81 .5 1 83 .1 0 Sh ar eG PT 4V 84 .0 8 1. 82 ↓ 82 .2 6 98 .2 0 98 .2 0 59 .4 9 10 .9 9 ↓ 4 8. 5 0 76 .3 0 84 .2 0 56 .7 9 9. 52 ↓ 47 .2 7 79 .3 0 86 .3 0 83 .8 4 40 .0 6 ↓ 4 3. 7 8 55 .2 9 55 .6 6 C og V L M 85 .9 7 0. 99 ↓ 84 .9 8 96 .7 4 99 .0 3 65 .8 7 2. 22 ↓ 63 .6 5 88 .1 6 96 .7 7 64 .2 4 2. 24 ↓ 62 .0 0 92 .2 7 96 .7 8 82 .3 2 5. 01 ↓ 7 7. 3 1 93 .1 8 94 .4 7 m PL U G -O w l 26 .3 1 0. 30 ↓ 26 .0 1 96 .2 5 99 .1 4 46 .9 0 5. 76 ↓ 41 .1 4 70 .8 9 91 .1 2 35 .9 3 2. 40 ↓ 33 .5 3 73 .7 3 94 .7 0 43 .2 5 6. 30 ↓ 3 6. 9 5 78 .7 7 90 .8 8 m PL U G -O w l2 77 .8 6 1. 21 ↓ 76 .6 5 98 .7 3 98 .7 3 54 .7 0 9. 22 ↓ 45 .4 8 78 .6 1 87 .0 3 50 .9 9 9. 02 ↓ 41 .9 7 79 .0 6 84 .9 0 56 .6 8 12 .1 4 ↓ 4 4. 5 4 82 .8 6 83 .9 4 Q w en -V L -C ha t 91 .6 6 2. 50 ↓ 89 .1 6 97 .3 5 97 .3 5 62 .3 2 14 .4 5 ↓ 4 7. 8 7 68 .4 2 77 .6 7 66 .5 6 15 .3 1 ↓ 51 .2 5 71 .7 7 77 .6 1 88 .0 1 46 .6 6 ↓ 4 1. 3 5 46 .9 0 48 .1 9 O tte r 62 .4 7 0. 91 ↓ 61 .5 6 98 .4 2 98 .9 1 53 .5 5 18 .7 1 ↓ 3 4. 8 4 58 .6 1 72 .1 9 46 .5 2 10 .5 1 ↓ 36 .0 1 66 .9 0 81 .4 9 45 .6 8 26 .0 2 ↓ 1 9. 6 6 60 .4 7 70 .1 0 ID E FI C S 44 .2 8 1. 29 ↓ 42 .9 9 97 .9 5 98 .6 3 42 .4 6 8. 69 ↓ 33 .7 7 75 .5 7 87 .4 7 32 .7 0 5. 46 ↓ 27 .2 4 83 .0 4 92 .4 1 43 .8 5 13 .8 8 ↓ 2 9. 9 7 81 .3 1 85 .6 4 In st ru ct B L IP 76 .0 4 1. 06 ↓ 74 .9 8 98 .7 0 98 .7 0 58 .7 8 8. 78 ↓ 50 .0 0 77 .6 8 87 .3 3 54 .7 2 5. 30 ↓ 49 .4 2 86 .2 3 91 .8 3 49 .3 9 3. 72 ↓ 4 5. 6 7 94 .6 2 95 .3 9 13 B LL M L L aV A -1 .5 -1 3B 81 .2 7 1. 67 ↓ 79 .6 0 98 .3 2 98 .3 2 60 .2 8 10 .9 0 ↓ 4 9. 3 8 77 .2 1 86 .1 8 57 .7 0 6. 87 ↓ 50 .8 3 83 .6 4 90 .5 3 87 .4 8 40 .1 4 ↓ 4 7. 3 4 57 .2 4 57 .3 3 Sh ar eG PT 4V -1 3B 84 .3 1 1. 36 ↓ 82 .9 5 98 .5 6 98 .5 6 60 .4 6 8. 51 ↓ 51 .9 5 78 .7 4 88 .8 6 58 .5 3 7. 04 ↓ 51 .4 9 83 .4 5 89 .8 2 83 .5 4 35 .5 8 ↓ 4 7. 9 6 59 .8 5 60 .2 2 In st ru ct B L IP -1 3B 71 .1 1 10 .0 1 ↓ 6 1. 1 0 85 .1 8 89 .6 6 49 .2 9 9. 40 ↓ 39 .8 9 67 .2 7 85 .9 7 47 .0 2 8. 69 ↓ 38 .3 3 71 .3 0 87 .1 5 45 .5 2 22 .0 8 ↓ 2 3. 4 4 74 .8 3 78 .3 3 C lo se dSo ur ce G PT -4 V 90 .1 4 85 .5 2 ↓ 4. 62 10 .0 1 10 .2 6 48 .3 2 39 .8 0 ↓ 8. 52 21 .8 3 31 .5 6 41 .6 4 34 .1 9 ↓ 7 .4 5 21 .8 7 32 .6 0 58 .5 0 45 .2 2 ↓ 1 3. 2 8 45 .7 8 48 .6 4
Ta bl
e 6:
Ex pe
rim en
tr es
ul ts
fo rm
od el
s an
sw er
in g
th e
sa m
e qu
es tio
ns un
de rt
w o
di ff
er en
tc on
ve rs
at io
n se
tti ng
s: C
le an
C on
v. an
d H
al lu
C on
v. N
um be
rs th
at ar
e hi
gh lig
ht ed
or an ge re pr es en tt he m od el pe rf or m an ce dr op ca us ed by ha llu ci na to ry co nv er sa tio n, co m pa re d to th e C le an C on v. A ll ex pe ri m en ts ar e im pl em en te d un de ra ze ro -s ho ts et tin g to av oi d th e bi as in tr od uc ed by de m on st ra tio ns .N ot e th at al lt he m od el s ex ce pt G PT -4 V ha ve a si gn ifi ca nt dr op in pe rf or m an ce on H al lu C on v. ,a nd th is tr en d ho ld s fo rG PT -4 V as w el l.
CleanConv., FactConv., IrrConv., and HalluConv. settings. Numbers that are highlighted orange and green represent the model performance drop and improvement in different conversation settings, compared to the model performance under CleanConv. setting.
ground answers are highlighted green and the hallucinated answers are highlighted red .

heading: Future Work
text: However, our methods still have limitations when deployed to a generalpurpose assistant, which we left as future works. 6 Limitations
In this work, with a carefully designed evaluation framework, we have revealed that current LVLMs severely suffer from multimodal hallucination snowballing. We further proposed the RVD to mitigate the phenomenon. However, our work still has limitations. Firstly, despite the greater variety of hallucination snowballing phenomena in the realworld setting, the scenarios we focus on are still relatively simplistic. This is because constructing rich and diverse scenarios would be more difficult and would require a significant amount of effort. Secondly, instead of meticulously finding real hallucinations generated by each LVLM and constructing relevant question-answer pairs, we choose to conduct experiments on our simulated hallucinatory conversations. This is because the evaluation processes based on responses from a single LVLM will make it difficult to scale up the evaluation data and adapt to more models. Thirdly, our experiments are conducted on models of 7B and 13B sizes, and we evaluate our proposed RVD only on a few selected models. This is due to computational limitations. Fourthly, our proposed RVD is currently still limited in several conversation scenarios. We will further explore expanding this method to more diverse conversation scenarios. 7 Acknowledgments
Xiaocheng Feng is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was supported by the National Key R&D Program of China via grant No. 2021ZD0112905, National Natural Science Foundation of China (NSFC) via grant (62276078, U22B2059), the Key R&D Program of Heilongjiang via grant 2022ZX01A32, the International Cooperation Project of PCL, PCL2022D01 and the Fundamental Research Funds for the Central Universities via grant No. HIT. OCEF. 2023018. References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304. 13734. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile visionlanguage model for understanding, localization, text reading, and beyond. Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023a. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310. 09478. Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023b. Sharegpt4v: Improving large multimodal models with better captions. arXiv preprint arXiv:2311. 12793. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023c. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An opensource chatbot impressing gpt-4 with 90%* chatgpt quality. Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic analysis of hallucination in gpt-4v(ision): Bias and interference challenges. W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards generalpurpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305. 06500. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305. 04790. Anisha Gunjal, Jihan Yin, and Erhan Bas. 2023. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308. 06394. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311. 05232. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207. 05608. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR). Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s "up" with vision-language models? investigating their struggle with spatial reasoning. Haoqiang Kang, Juntong Ni, and Huaxiu Yao. 2023. Ever: Mitigating hallucination in large language models through real-time verification and rectification. arXiv preprint arXiv:2311. 09114. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73. Hugo Laurençon and Daniel van Strien. 2023. Introducing idefics: An open reproduction of state-of-the-art visual langage model. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. Mitigating object hallucinations in large visionlanguage models through visual contrastive decoding. arXiv preprint arXiv:2311. 16922. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305. 03726. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105. 03023. Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023a. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v(ision), llava-1. 5, and other multi-modality models. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023b. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306. 14565, 1(2):9. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402. 00253. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023c. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310. 03744. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023d. Visual instruction tuning. arXiv preprint arXiv:2304. 08485. Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023e. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311. 05437. Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv preprint arXiv:2310. 05338. Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. 2023. Evaluation and mitigation of agnosia in multimodal large language models. OpenAI. 2022. Introducing chatgpt. Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. Yunjie Tian, Tianren Ma, Lingxi Xie, Jihao Qiu, Xi Tang, Yuan Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. 2024. Chatterbox: Multi-round multimodal referring and grounding. arXiv preprint arXiv:2401. 13307. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971. Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023a. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311. 07397. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023b. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311. 03079. Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2023. The generative ai paradox: "what it can create, it may not understand". Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE. Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth KY Wong, Zhenguo Li, and Hengshuang Zhao. 2023. Drivegpt4: Interpretable end-toend autonomous driving via large language model. arXiv preprint arXiv:2310. 01412. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023a. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023b. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311. 04257. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023c. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311. 04257. Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310. 16045. Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. 2023. Halle-switch: Controlling object hallucination in large vision language models. arXiv e-prints, pages arXiv–2310. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023a. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534. Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023b. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 15112. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 18223. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401. 01614. Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310. 00754. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592. Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. 2023. Fool your (vision and) language model with embarrassingly simple permutations. ArXiv preprint, abs/2310. 01651. A Additional Experimental Details
A. 1 Hallucination Allocation After carefully analyzing the question-answer pairs in the dataset, we manage to create an answer vocabulary for answers suitable for introducing relation errors. What’s more, we utilize Part-ofSpeech2 of the answer in the fact sentence to choose proper hallucination types. Specifically, we allocate attribute hallucination for those answers tagged as adjectives and verbs and allocate existence hallucination for those answers tagged as nouns. For imagination hallucination, instead of using the annotated question-answer pair, we provide ChatGPT with all annotated objects and ask ChatGPT to generate an object that is not present in the image but is reasonable to be in the corresponding scene. Then, we directly construct a question-answer pair with the template: "question: Is there a in the image? answer: No". A. 2 Prompts In this section, we list all prompts used during the process of constructing hallucinatory conversations, which include fact generation (Figure 9), conflict creation (Figure 10), description generation (Figure 11) and conflict verification (Figure 12). Note that we reuse the description generation prompt (Figure 11) to generate the ground image description by giving the annotated regional description and fact sentence. A. 3 Manual Checking We randomly select 100 data for each hallucination type in our curated dataset, 400 in total. We ask three annotators to check each of them from three aspects, as Table 5 depicts. The annotation results show that the generated hallucinatory description mostly meets our requirements. A. 4 Generation Details Through all our experiments, we follow a consistent generation configuration to ensure fairness. Specifically, we set the inference hyperparameter as follows: do_sample=True, temperature=1. 0, top_p=0. 95, top_k=None and num_beams=1. A. 5 Hallucinatory Conversation Example We list four examples to demonstrate curated hallucinatory conversation of each hallucination type, namely existence(Figure 13), attribute(Figure 14),
2We use Spacy to do the Part-of-Speech tagging. relation(Figure 15), and imagination(Figure 16), where the ground answers are highlighted green and the hallucinated answers are highlighted red