Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024)

Abstract: Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to 4% improvement in POPE F1 scores and up to 36% reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms for improved multimodal performance.

heading: 1 Introduction
text: Large Vision-Language Models (LVLMs) are a multimodal extension of Large Language Models (LLMs), transforming textual prompts and image inputs into text. However, they frequently produce object hallucinations, where absent objects are mentioned in the output (Yifan Li and Wen, 2023; Lovenia et al., 2023).
While hallucination-mitigation techniques in LLMs are actively researched, specific strategies for LVLMs are less developed. Current methods involve model-specific adjustments, additional training, or auxiliary models for post-hoc correction, and are often proven inefficient, costly, or limited by training data and model biases (Wang et al., 2023; Zhou et al., 2023; Gunjal et al., 2023; Yin
et al., 2023). Conversely, LVLM hallucination evaluation has seen progress with object hallucination benchmarks like NOPE (Lovenia et al., 2023) and POPE (Yifan Li and Wen, 2023), and recent works that aim for more holistic LVLM hallucination evaluation such as FaithScore (Jing et al., 2023) and HallusionBench (Guan et al., 2023).
A key reason for LVLM hallucinations is their tendency to over-rely on linguistic information, as was first observed by Guan et al. (2023). Based on this insight, we propose to intervene in the LVLM decoding phase so that model outputs are less informed by language biases. Specifically, we propose to use Contrastive Decoding (Li et al., 2023a;
6008
O’Brien and Lewis, 2023) to alter LVLM output probabilities with respect to the internal LLM’s probabilities, guided by a dynamic weighting mechanism based on the LLM distribution’s entropy.
Our experiments show that our proposed method, Language Contrastive Decoding (LCD), improves hallucination scores on POPE (Yifan Li and Wen, 2023) and CHAIR (Rohrbach et al., 2018) on InstructBLIP variants based on Vicuna and Flan-T5 (Dai et al., 2023), LLAVA 1.5 (Liu et al., 2023) and mPLUG-Owl2 (Ye et al., 2023). We asses LCD’s overall generation quality by reporting captioning metrics and conducting a GPT4-V (OpenAI et al., 2023) assisted evaluation. LCD, as a decoding strategy, can be applied to other models without additional training or output modifications, emphasizing its utility for broader LVLM use.
The contributions of this paper are thus manifold. First, we introduce a novel decoding method tailored for LVLMs to mitigate object hallucinations. Next, we present a dynamic weighting strategy based on entropy which is applicable in various CD scenarios. Finally, we share our code to encourage further research into LVLM-specific decoding strategies, a promising avenue for future research.

heading: 2 Motivation and Background
text: The integration of vision capabilities into LLMs has led to the development of Large VisionLanguage Models, merging LLMs’ textual understanding with vision-text encoders. This trend towards multimodal systems is exemplified in commercial platforms such as GPT4-V (OpenAI et al., 2023) and Google’s Gemini (Team et al., 2023).
Large Vision-Language Models combine LLMs and vision-text encoders to generate text from textual prompts and visual inputs. An LVLM generally comprises three main components: a vision-text encoder like CLIP (Radford et al., 2021), an LLM such as LLAMA (Touvron et al., 2023) or Flan-T5 (Chung et al., 2022), and a cross-modal alignment module linking the vision-text encoder output with the LLM.
Initially, LVLMs were fine-tuned for specific tasks (Li et al., 2022; Wang et al., 2022). However, advancements in LLMs have led to a shift towards general-purpose, instruction-tuned LVLMs. These models are designed to handle a wide range of tasks based on instructions, making them more versatile. Despite these advancements, LVLMs grapple with hallucinations of different types.
LVLMs Hallucinations and their Mitigation Hallucinations in LVLMs, particularly object hallucinations where nonexistent entities are mentioned, are often attributed to LVLMs’ reliance on spurious correlations and language biases, as demonstrated by Li et al. (2023b) and Zhou et al. (2023). Moreover, Guan et al. (2023) highlight LVLMs’ tendency to prioritize language over visual data, leading to hallucinations.
Mitigation strategies proposed by Gunjal et al. (2023) and Wang et al. (2023) involve further model training with augmented datasets or reward models. Zhou et al. (2023); Yin et al. (2023) developed auxiliary models to correct outputs postgeneration. These solutions often require datasetspecific work or additional model training, potentially leading to overfitting or new biases, and are not easily transferable across LVLMs.
In a concurrent work, Leng et al. (2023) develop an LVLM-specific decoding algorithm for mitigating hallucinations, using a noisy copy of the input image as a contrastive input. While their approach uses visual noise to guide the decoding process, LCD leverages the language modality to mitigate hallucinations. These approaches are orthogonal and can potentially be combined into a unified Language-Visual contrastive decoding algorithm, a direction we leave for future work. 1

heading: 3 Language Contrastive Decoding (LCD)
text: Before presenting LCD, we briefly introduce the essentials of decoding in LVLMs 3.1, followed by our formal proposal 3.2 and research hypothesis 3.3.

heading: 3.1 Decoding Techniques and Contrastive Decoding: Essential Preliminaries
text: Decoding in auto-regressive generative models is the stage that transforms an input representation into a sequence of output tokens. In LVLMs, this process involves a model M , an image I , a textual prompt X , and a particular timestamp t during generation. It can be described as a series of selections from the model’s probability distribution, producing a token sequence T , as formalized in eq. (1).
Tt ∼ P (·|I,X, T<t;M) (1) Greedy decoding, selecting the most probable
token at each step (or the top k tokens in a beam 1Favero et al. (2024) propose a method with a high resemblance to ours, however, our work predates theirs. https://openreview.net/forum?id=aReb-02mhR
search with beam size k), is the simplest approach. However, high likelihood sequences do not necessarily align with human preferences, leading to the “likelihood trap” (Zhang et al., 2021). This has led to the use of sampling-based methods, such as top-k sampling, nucleus sampling (Holtzman et al., 2020), and locally typical sampling (Meister et al., 2023), which either truncate the set of candidate tokens or adjust the model’s distribution, e.g. through temperature scaling.
Contrastive Decoding (CD) has been introduced for LLMs as a method to penalize the outputs of an expert model with those from a less powerful model (Li et al., 2023a). CD can be applied to any two probability distributions with the same support and has been adapted as a sampling strategy, improving various text generation tasks (O’Brien and Lewis, 2023; Chuang et al., 2023; Sennrich et al., 2024). CD uses both truncation and reshaping of probability distributions. The truncation phase ("adaptive plausibility") is described by eq. (2), where α is a hyper-parameter, V and Vt′ are the original and truncated token vocabularies at time t, and P is the conditional distribution on the prefix T<t.
Vt′ = {v ∈ V : P (v|T<t) ≥ αmax w P (w|T<t)} (2)
Finally, the formula for CD, as suggested by O’Brien and Lewis (2023), given here generally for two conditional distributions P and P ′ on variable x with the same support, conditioned on a prefix sequence X is presented in eq. (3).
CDt(x,X, P, P ′) = { (1 + β) logP (x|X)− β logP ′(x|X), if x ∈ V t′ −∞, otherwise (3)
β is a fixed weight hyper-parameter. Our proposed method, detailed shortly, alters CD by introducing an entropy-based dynamic weighting scheme.

heading: 3.2 Proposed Method
text: Our intuition, based on previous findings by (Guan et al., 2023; Rohrbach et al., 2018; Yifan Li and Wen, 2023), is that an LVLM can be "misled" by its constituent LLM during the generation process.
Consider for example an LVLM that is describing an image (see illustration 1). Mid-generation, given the text "An image of a man walking his," it may predict "dog" due to language biases, even if it is a bear that is actually shown. A ‘plain’ LLM,
without seeing the image, reinforces these biases by highly rating “dog”. Our method builds on this insight to guide an LVLM towards more accurate predictions using Contrastive Decoding.
Our method operates as follows: At each generation step t, for each token x, we first determine the next-token probabilities from the LVLM, PLV LM , based on the current token sequence T<t, text X , and image I . We then obtain a second distribution, PLLM , by inputting all data except the image into the LLM. The LLM’s conditional entropy HLLM informs the dynamic weight as per eq.(4). We then adjust token x’s logits using the LCD formula in eq. (5).
βt = β
HLLM (x|X,T<t) (4)
LCDt(x, T<t, I, PLV LM , PLLM ) =
(1 + βt) logPLV LM (x|I,X, T<t) − βt logPLLM (x|X,T<t)
(5) In our experiments, we generate text completions by sampling from the next token probabilities, which are obtained by applying the softmax function to the logits produced by the LCD algorithm.

heading: 3.3 Research Hypothesis
text: Our hypothesis is that contrasting LVLM outputs with LLM outputs conditioned only on the textual data, can mitigate language biases, therefore reducing hallucinations in LVLMs.

heading: 4 Experiments and Results
text: We set out to assess the effect of LCD on object hallucinations in LVLM outputs against popular decoding settings. Additionally, we verify that LCD does not degrade output quality. To this end, we asses LCD on the POPE benchmark (Yifan Li and Wen, 2023), and on an image detailed-description task where we report hallucination and captioning metrics and conduct a GPT4-V assisted evaluation.
Polling-based Object-Probing Evaluation POPE consists of object-presence binary questions on 500 COCO dataset images (Lin et al., 2015), with questions equally divided between present and absent objects. Absent objects are chosen based on three criteria: random, popular (common in COCO), and adversarial (commonly co-occurring with present objects). POPE’s drawback is its
one-word response structure, which limits the influence of decoding strategies and does not evaluate open-ended generation capabilities.
Image Detailed-Descriptions To complement POPE, we introduce a long-form text generation task called "Image Detailed-Descriptions," inspired by findings from Zhou et al. (2023) that more extensive context increases the likelihood of hallucinations. In this task, the input consists of an image from the COCO dataset and a text prompt requesting a detailed description of the image. The expected output is a long-form, detailed textual description of the given image, typically containing multiple sentences. The prompts used in this task are detailed in appendix A.1. By using the same COCO images as POPE, we maintain consistency in the visual domain while exploring LCD’s effectiveness in a more challenging setting where the model is required to generate longer, more descriptive outputs.
Baselines and Metrics For POPE, we use sampling as the baseline and report F1 scores.2 For the detailed-descriptions task, we use as a baseline the popular nucleus sampling algorithm3 and report CHAIR metrics (Rohrbach et al., 2018). To assess description quality, we use captioning metrics against COCO’s gold captions, which serve as an approximation considering length differences. Additionally, following Yin et al. (2023), we use GPT4-V to evaluate the descriptions for Detailedness and Accuracy (see details in Appendix A.1).
Models We conduct our experiments with leading LVLMs: two versions of the InstructBLIP model (with Flan-T5 and Vicuna LLMs), LLAVA
2Complete POPE results are in the appendix, table 4 3We find that nucleus-sampling gives better results than
vanilla sampling (see table 3 in the appendix for ablations).
1.5 and mPLUG-Owl2. The complete experimental details, such as exact model variants and generation hyper-parameters, are given in the Appendix.

heading: 5 Results and Discussion
text: For the POPE task, which evaluates object hallucinations using binary questions, LCD improves F1 scores across 11 out of 12 configurations compared to the baseline (Table 2). This suggests that LCD is effective in reducing object hallucinations in the POPE setting. It is worth noting that the POPE setting is highly constrained for decoding algorithms, as it consists of binary yes/no questions, and typically involves only a single decoding step. This limits the potential impact of decoding strategies on the model’s performance in this specific task.
In the detailed-description task, which involves generating detailed descriptions of images, LCD significantly reduces hallucinations at both sentence and instance levels across all four models tested (Table 1). However, it is important to note that despite the improvements, the CHAIR scores, which measure hallucination rates (lower is better), remain relatively high. This indicates that object hallucinations are still prevalent in long-form LVLM outputs, even with the application of LCD.4
We observe that LCD is particularly effective in improving the performance of InstructBLIP models (InstructBLIPF and InstructBLIPV). We hypothesize that this may be due to the fact that the LLMs in these models are frozen during training, which results in a stronger language bias that LCD can effectively mitigate. When evaluating the overall generation quality using captioning metrics (METEOR, WMD, and ROUGEL), LCD outperforms the baseline in all cases except one (WMD in
4Examples of generated descriptions are found in Appendix A.2
LLAVA 1.5, where the reduction is approximately 1%). This indicates that LCD not only reduces hallucinations but also maintains or improves the overall quality of the generated descriptions.
Furthermore, in the GPT4-V assisted evaluation, which assesses the accuracy and detailedness of the generated descriptions, LCD improves the accuracy scores across all models. Interestingly, the detailedness scores remain similar to the baseline, suggesting that LCD reduces hallucinations without increasing the granularity of the descriptions.

heading: 6 Conclusion
text: In this paper we present Language Contrastive Decoding, a novel method to reduce hallucinations in LVLMs. By dynamically adjusting output probabilities using the LVLM’s internal LLM, LCD significantly improves hallucination metrics across different LVLM architectures, enhancing the quality and reliability of generated content without necessitating retraining or auxiliary models and postprocessing. This work highlights the potential of specialized decoding strategies in enhancing multimodal AI models and lays the groundwork for further exploration into more sophisticated LVLM decoding methods.

heading: 7 Limitations
text: Firstly, while LCD shows promise in reducing hallucinations, it only targets hallucinations caused by language biases, but hallucinations can arise from other sources. For instance, previous work has shown that some hallucinations are caused by poor visual understanding (Guan et al., 2023). We believe LCD can be used as a platform to craft LVLM-specific decoding algorithms that would mitigate hallucinations stemming from different
factors, and leave this pursuit for future work. Secondly, our evaluation method primarily ad-
dresses object hallucinations, which are only one form of hallucination that LVLMs may exhibit. Preliminary results signal that LCD mitigates more complex manifestations of language-induced hallucinations as assessed by recent benchmarks such as FAITHSCORE (Jing et al., 2023) and HallusionBench (Guan et al., 2023), but further work is required to establish this.
Moreover, LCD relies on current LVLM architectures that combine an LLM and a text-vision encoder, and requires access to an LLM that emits output probabilities on the same set of tokens as the LVLM. It is possible that the future generation of multimodal AI systems will have a different architecture that will make LCD obsolete. Additionally, LCD requires an LLM forward pass for each LVLM decoding step. The added latency could be mitigated with efficient inference techniques, and also by using a smaller LLM as the contrasting model. The effectiveness of LCD in this scenario is left for future work.
Finally, there are ethical considerations related to the mitigation of hallucinations in LVLMs. As these models become more reliable, it is crucial to continue evaluating the potential impacts of their use, ensuring they do not perpetuate or exacerbate biases present in their training data. LCD indeed mitigates some biases, but it is important to keep in mind that it might amplify other biases, unknown to us. Responsible deployment of these models requires ongoing vigilance and a commitment to transparency and fairness.

heading: Acknowledgements
text: We thank Yoav Goldberg, Ido Dagan, and the participants of the NLP seminar at Bar-Ilan University for their valuable feedback. This research has been funded by a grant from the European Research Council, ERC-StG grant number 677352, and a grant by the Israeli Science Foundation (ISF), grant number 670/23, for which we are grateful.

heading: A Appendix
text: A.1 Detailed Experimental Setup For POPE and the descriptions experiment, we use the following LCD parameters β = 3.0, α = 0.1. We set the temperature to 0.5 in POPE and 1.0 in the descriptions experiment. We limit the descriptions length to 250 tokens in all models we tested. We don’t tune any of these parameters. The prompt we use for the descriptions experiment is "Describe this image in detail:". The models we use have the following Huggingface identifiers:
• Salesforce/instructblip-vicuna-7b
• Salesforce/instructblip-flan-t5-xl
• llava-hf/llava-1.5-7b-hf
• MAGAer13/mplug-owl2-llama2-7b
GPT4-V Assisted Evaluation We follow the evaluation protocol given in Yin et al. (2023), where an image and two descriptions are given to the model, formatted with the prompt in figure 2. The model outputs scores in two dimensions: Accuracy and Detailedness. We use the gpt-4-vision-preview model on February 2024.
A.2 COCO Detailed Descriptions Examples The descriptions in this section were generated by the LLAVA 1.5 model.
COCO Image 461331 Nucleus Sampling: The image features a white counter area with various appliances and a sign. On top of the counter, there is a microwave positioned alongside a toaster oven. Below the toaster oven is a sign with suggestions to keep people safe while using the kitchen. Several appliances are present, including two cups, one being closer to the foreground, and the other near the edge of the counter. A sink is located near the lower left corner, and a spoon can be seen towards the right side of the counter. There are two ovens, with one situated towards the right side, and the other near the center of the counter. Lastly, a bottle is visible in the background, at the far left edge. LCD: The image showcases a kitchen setting featuring a microwave sitting on top of a shelf. Below the microwave, a toaster oven is placed in close proximity. There are also a couple of toaster ovens situated further back in the scene. A note is posted on the shelf, offering suggestions to keep everyone safe by covering one’s experience while using the appliances, cleaning up after themselves, and not turning on the microwaves with no one around. The scene reflects a cooperative environment for kitchen users.
COCO Image 378873 Nucleus Sampling: The image features an outdoor market with a variety of fruits and vegetables for sale. Two people can be seen in the scene, likely browsing the options and making their selections. The produce consists of several bunches of ripe red tomatoes, and smaller, freshly picked green peppers. There is also a larger selection of yellow bell peppers available in various sizes. In addition to the vegetables, there is a bottle visible near the top left corner. Overall, the market offers an array of fresh, colorful, and healthy products for customers to choose from. LCD: The image showcases an outdoor produce market filled with an abundant selection of fruits and vegetables. Numerous tomatoes can be found throughout the market, in various locations and sizes. Peppers are another prominent feature at this market, with multiple bunches spread across different parts of the display. A person is visible near the top left of the scene, likely browsing through the market offerings or engaging with the sellers. Overall, it’s a lively marketplace filled with fresh and healthy options.
COCO Image 498100 Nucleus Sampling: The image displays a room filled with numerous decorative items. A prominent green ceramic vase stands out, with a colorful art deco sculpture depicting a woman laying on it. There are also two smaller vases in the scene – one in the upper right corner and the other closer to the center of the room. The room is adorned with a variety of objects, including a clock on the wall, a collection of books scattered in the background, and a chair positioned in the middle. This visually rich space showcases a diverse assortment of vases, sculptures, and other home decor items. LCD: The image showcases an assortment of vases on display, featuring a variety of colors and sizes. Among these, there is a large green vase as the focal point of the scene. Inside this large green vase, a statue of a woman can be seen lying on the side of the vase. Another green vase is located towards the right side of the scene. Additionally, a smaller green vase is found close to the center. Two other smaller vases can also be spotted within the collection. The diverse display of vases creates an engaging and artistic atmosphere for visitors to enjoy.
COCO Image 312406 Nucleus Sampling: The image features a tray of cake covered with foil and a plastic container with coconut powder, all placed on a counter. There are several sliced chocolate cakes, some of which are stacked. Additionally, a pair of tongs is resting beside the cakes, ready to be used for serving. A bowl with a spoon in it is also situated on the table, accompanied by a banana and a spoon nearby. The overall arrangement looks like a setting for a delicious dessert. LCD: The image features a dining table with various items arranged on it. A banana can be seen prominently, occupying a large portion of the left side of the table. Along with the banana, there’s a can of powdered cocoa placed towards the top-middle area of the table. Additionally, two bowls are present on the table – one occupying the middle-right part and another covering a smaller section on the right side. There is also a spoon situated on the right side of the table, ready for use in enjoying the snack or dessert.
COCO Image 379404 Nucleus Sampling: The image displays a delicious meal consisting of chicken, broccoli, and a banana. The plate is large and fills most of the frame, with various items on the plate. The chicken is located at the bottom-right side of the plate, while the broccoli pieces are scattered throughout the plate. Additionally, there is a banana on the plate that is placed towards the top left corner. In the background, a bottle can be seen, likely containing a drink that complements the meal. LCD: The image features a white plate topped with various foods. On the plate, there’s a banana positioned towards the left, accompanied by some meat, likely fish sticks. Alongside the main items, a glass of skim milk is present, adding to the nutritious meal. There is also a sandwich, possibly a piece of bread, contributing to the diverse meal displayed on the plate.
COCO Image 125524 Nucleus Sampling: The image features a bright red fire hydrant prominently placed on the side of the street. The fire hydrant is situated close to a bus stop and appears to be unused. A large white bus is positioned behind the fire hydrant on the other side of the street. A car is also visible in the background, slightly to the right. In addition, there are two people in the scene, both located at the far right side. One of them is holding a cell phone, possibly capturing the moment or checking their phone. LCD: In the scene, a red fire hydrant is situated next to the street, capturing attention due to its striking color. A bus is driving down the road nearby, positioned closer to the background of the image. There are several people dispersed along the street. Some of them can be seen closer to the middle and right side of the scene, while others are scattered across the entire street, possibly going about their day or waiting to board the bus.
A.3 Detailed Experimental Results

heading: Future Work
text: We believe LCD can be used as a platform to craft LVLM-specific decoding algorithms that would mitigate hallucinations stemming from different
factors, and leave this pursuit for future work. Secondly, our evaluation method primarily ad-
dresses object hallucinations, which are only one form of hallucination that LVLMs may exhibit. Preliminary results signal that LCD mitigates more complex manifestations of language-induced hallucinations as assessed by recent benchmarks such as FAITHSCORE (Jing et al. , 2023) and HallusionBench (Guan et al. , 2023), but further work is required to establish this. Moreover, LCD relies on current LVLM architectures that combine an LLM and a text-vision encoder, and requires access to an LLM that emits output probabilities on the same set of tokens as the LVLM. It is possible that the future generation of multimodal AI systems will have a different architecture that will make LCD obsolete. Additionally, LCD requires an LLM forward pass for each LVLM decoding step. The added latency could be mitigated with efficient inference techniques, and also by using a smaller LLM as the contrasting model. The effectiveness of LCD in this scenario is left for future work. Finally, there are ethical considerations related to the mitigation of hallucinations in LVLMs. As these models become more reliable, it is crucial to continue evaluating the potential impacts of their use, ensuring they do not perpetuate or exacerbate biases present in their training data. LCD indeed mitigates some biases, but it is important to keep in mind that it might amplify other biases, unknown to us. Responsible deployment of these models requires ongoing vigilance and a commitment to transparency and fairness