Title: Model-aware Counterfactual Data based Contrastive Decoding for Video-LLM
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video-language models, Hallucination mitigation, Contrastive decoding, Object-level data augmentation, Counterfactual inputs

Abstract: Video language models (Video-LLMs) are prone to hallucinations, often generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing decoding methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for mitigating hallucination patterns. However, such a way is hard to control the visual cues that drive hallucination or well align with model weaknesses. 
We propose Model-aware Counterfactual Data based Contrastive Decoding (GeWu), a new inference strategy that combines model-guided counterfactual construction with decoding. Our approach uses the Video-LLM’s own feedback to identify object regions most responsible for hallucination, generating targeted counterfactual inputs at the object level rather than arbitrary frame or temporal modifications. These model-aware counterfactual data is then integrated into CD to enforce evidence-grounded token selection during decoding. Experiments on EventHallusion, MVBench, and Perception-test show that GeWu consistently reduces hallucination while maintaining or improving task accuracy across diverse Video-LLMs, including InternVL3, Qwen2.5-VL and Qwen2-VL families. The method is especially effective in challenging scenarios involving small, occluded, or co-occurring objects. Our code and data will be publicly released.