VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning
  for Image and Video Generation
Authors: name: Jiazheng Xu; name: Yu Huang; name: Jiale Cheng; name: Yuanming Yang; name: Jiajun Xu; name: Yuan Wang; name: Wenbo Duan; name: Shen Yang; name: Qunlin Jin; name: Shurun Li; name: Jiayan Teng; name: Zhuoyi Yang; name: Wendi Zheng; name: Xiao Liu; name: Ming Ding; name: Xiaohan Zhang; name: Xiaotao Gu; name: Shiyu Huang; name: Minlie Huang; name: Jie Tang; name: Yuxiao Dong
Published: 2024-12-30 16:24:09

Abstract: We present a general strategy to aligning visual generation models -- both
image and video generation -- with human preference. To start with, we build
VisionReward -- a fine-grained and multi-dimensional reward model. We decompose
human preferences in images and videos into multiple dimensions, each
represented by a series of judgment questions, linearly weighted and summed to
an interpretable and accurate score. To address the challenges of video quality
assessment, we systematically analyze various dynamic features of videos, which
helps VisionReward surpass VideoScore by 17.2% and achieve top performance for
video preference prediction. Based on VisionReward, we develop a
multi-objective preference learning algorithm that effectively addresses the
issue of confounding factors within preference data. Our approach significantly
outperforms existing image and video scoring methods on both machine metrics
and human evaluation. All code and datasets are provided at
https://github.com/THUDM/VisionReward.