Aligning Multimodal LLM with Human Preference: A Survey
Authors: name: Tao Yu, name: Yi-Fan Zhang, name: Chaoyou Fu, name: Junkang Wu, name: Jinda Lu, name: Kun Wang, name: Xingyu Lu, name: Yunhang Shen, name: Guibin Zhang, name: Dingjie Song, name: Yibo Yan, name: Tianlong Xu, name: Qingsong Wen, name: Zhang Zhang, name: Yan Huang, name: Liang Wang, name: Tieniu Tan

Large language models (LLMs) can handle a wide variety of general tasks with
simple prompts, without the need for task-specific training. Multimodal Large
Language Models (MLLMs), built upon LLMs, have demonstrated impressive
potential in tackling complex tasks involving visual, auditory, and textual
data. However, critical issues related to truthfulness, safety, o1-like
reasoning, and alignment with human preference remain insufficiently addressed.
This gap has spurred the emergence of various alignment algorithms, each
targeting different application scenarios and optimization goals. Recent
studies have shown that alignment algorithms are a powerful approach to
resolving the aforementioned challenges. In this paper, we aim to provide a
comprehensive and systematic review of alignment algorithms for MLLMs.
Specifically, we explore four key aspects: (1) the application scenarios
covered by alignment algorithms, including general image understanding,
multi-image, video, and audio, and extended multimodal applications; (2) the
core factors in constructing alignment datasets, including data sources, model
responses, and preference annotations; (3) the benchmarks used to evaluate
alignment algorithms; and (4) a discussion of potential future directions for
the development of alignment algorithms. This work seeks to help researchers
organize current advancements in the field and inspire better alignment
methods. The project page of this paper is available at
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Alignment.