MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large
  Vision-Language Models
Authors: name: Ziyu Liu, name: Yuhang Zang, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Cao, name: Haodong Duan, name: Conghui He, name: Yuanjun Xiong, name: Dahua Lin, name: Jiaqi Wang

Visual preference alignment involves training Large Vision-Language Models
(LVLMs) to predict human preferences between visual inputs. This is typically
achieved by using labeled datasets of chosen/rejected pairs and employing
optimization algorithms like direct preference optimization (DPO). Existing
visual alignment methods, primarily designed for single-image scenarios,
struggle to effectively handle the complexity of multi-image tasks due to the
scarcity of diverse training data and the high cost of annotating
chosen/rejected pairs. We present Multi-Image Augmented Direct Preference
Optimization (MIA-DPO), a visual preference alignment approach that effectively
handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse
multi-image training data by extending single-image data with unrelated images
arranged in grid collages or pic-in-pic formats, significantly reducing the
costs associated with multi-image data annotations. Our observation reveals
that attention values of LVLMs vary considerably across different images. We
use attention values to identify and filter out rejected responses the model
may have mistakenly focused on. Our attention-aware selection for constructing
the chosen/rejected pairs without relying on (i) human annotation, (ii) extra
data, and (iii) external models or APIs. MIA-DPO is compatible with various
architectures and outperforms existing methods on five multi-image benchmarks,
achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the
recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's
ability to understand single images.