Improving Video Generation with Human Feedback
Authors: name: Jie Liu; name: Gongye Liu; name: Jiajun Liang; name: Ziyang Yuan; name: Xiaokun Liu; name: Mingwu Zheng; name: Xiele Wu; name: Qiulin Wang; name: Wenyu Qin; name: Menghan Xia; name: Xintao Wang; name: Xiaohong Liu; name: Fei Yang; name: Pengfei Wan; name: Di Zhang; name: Kun Gai; name: Yujiu Yang; name: Wanli Ouyang
Published: 2025-01-23 18:55:41

Abstract: Video generation has achieved significant advances through rectified flow
techniques, but issues like unsmooth motion and misalignment between videos and
prompts persist. In this work, we develop a systematic pipeline that harnesses
human feedback to mitigate these problems and refine the video generation
model. Specifically, we begin by constructing a large-scale human preference
dataset focused on modern video generation models, incorporating pairwise
annotations across multi-dimensions. We then introduce VideoReward, a
multi-dimensional video reward model, and examine how annotations and various
design choices impact its rewarding efficacy. From a unified reinforcement
learning perspective aimed at maximizing reward with KL regularization, we
introduce three alignment algorithms for flow-based models by extending those
from diffusion models. These include two training-time strategies: direct
preference optimization for flow (Flow-DPO) and reward weighted regression for
flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
reward guidance directly to noisy videos. Experimental results indicate that
VideoReward significantly outperforms existing reward models, and Flow-DPO
demonstrates superior performance compared to both Flow-RWR and standard
supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
weights to multiple objectives during inference, meeting personalized video
quality needs. Project page: https://gongyeliu.github.io/videoalign.