Title: The Last Token is Enough: Lightweight Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Trustworthy AI, Hallucination, Large Vision-Language Models, Multimodal large language model

Abstract: Contrastive decoding is a mainstream approach for mitigating hallucinations, which actively induces hallucinations and compares them with the original output to calibrate the logits of the next token, thereby enhancing the response reliability of LVLMs. Although contrastive decoding has shown promising effectiveness in suppressing hallucinations, the process of inducing hallucinations inevitably incurs additional computational overhead and considerable inference latency, an issue largely overlooked in existing studies. To this end, we present LightCD, a lightweight and efficient contrastive decoding method that mitigate hallucinations in LVLMs while reducing computational overhead and inference latency. Specifically, we observe that LVLMs tend to aggregate learned information into a small number of tokens, which play a critical role in supporting the final output. The last token, used for next-token prediction, leverages the attention mechanism to “see” and summarize this information in order to produce the final result. In light of this, we revisit the process of inducing hallucinations in the current contrastive decoding paradigm and propose processing solely on the last token in the final layer of LVLMs to replace full-sequence inference. LightCD introduces two key modules: Selective Attention Perturbation, which identifies and filters critical information from attention heads containing factual evidence; and a Residual-Free Mechanism, which suppresses cross-layer information propagation to generate desired hallucinations. Finally, the model completes contrastive decoding by subtracting the hallucination logits from the original token logits. Extensive experiments demonstrate that LightCD significantly outperforms existing methods in mitigating hallucinations while achieving a 2$\times$ increase in inference speed.