Preference Tuning with Human Feedback on Language, Speech, and Vision
  Tasks: A Survey
Authors: name: Genta Indra Winata; name: Hanyang Zhao; name: Anirban Das; name: Wenpin Tang; name: David D. Yao; name: Shi-Xiong Zhang; name: Sambit Sahu
Published: 2024-09-17 21:28:51

Abstract: Preference tuning is a crucial process for aligning deep generative models
with human preferences. This survey offers a thorough overview of recent
advancements in preference tuning and the integration of human feedback. The
paper is organized into three main sections: 1) introduction and preliminaries:
an introduction to reinforcement learning frameworks, preference tuning tasks,
models, and datasets across various modalities: language, speech, and vision,
as well as different policy approaches, 2) in-depth examination of each
preference tuning approach: a detailed analysis of the methods used in
preference tuning, and 3) applications, discussion, and future directions: an
exploration of the applications of preference tuning in downstream tasks,
including evaluation methods for different modalities, and an outlook on future
research directions. Our objective is to present the latest methodologies in
preference tuning and model alignment, enhancing the understanding of this
field for researchers and practitioners. We hope to encourage further
engagement and innovation in this area.