OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
Authors: name: Xiangyu Zhao; name: Shengyuan Ding; name: Zicheng Zhang; name: Haian Huang; name: Maosong Cao; name: Weiyun Wang; name: Jiaqi Wang; name: Xinyu Fang; name: Wenhai Wang; name: Guangtao Zhai; name: Haodong Duan; name: Hua Yang; name: Kai Chen
Published: 2025-02-25 18:05:14

Abstract: Recent advancements in open-source multi-modal large language models (MLLMs)
have primarily focused on enhancing foundational capabilities, leaving a
significant gap in human preference alignment. This paper introduces
OmniAlign-V, a comprehensive dataset of 200K high-quality training samples
featuring diverse images, complex questions, and varied response formats to
improve MLLMs' alignment with human preferences. We also present MM-AlignBench,
a human-annotated benchmark specifically designed to evaluate MLLMs' alignment
with human values. Experimental results show that finetuning MLLMs with
OmniAlign-V, using Supervised Fine-Tuning (SFT) or Direct Preference
Optimization (DPO), significantly enhances human preference alignment while
maintaining or enhancing performance on standard VQA benchmarks, preserving
their fundamental capabilities. Our datasets, benchmark, code and checkpoints
have been released at https://github.com/PhoenixZ810/OmniAlign-V.