Title: PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Multimodal LLMs, Alignment, Hallucination Reduction

Abstract: Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term \textbf{visual insensitivity}: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely \textbf{across-image insensitivity} and \textbf{within-image insensitivity}. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We empirically demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. We empirically evaluate PEA-DPO across three hallucination benchmarks using multimodal LLMs (MLLMs) of varying scales. Our results demonstrate that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.