Improving Video Generation with Human Feedback
Authors: name: Jie Liu, name: Gongye Liu, name: Jiajun Liang, name: Ziyang Yuan, name: Xiaokun Liu, name: Mingwu Zheng, name: Xiele Wu, name: Qiulin Wang, name: Wenyu Qin, name: Menghan Xia, name: Xintao Wang, name: Xiaohong Liu, name: Fei Yang, name: Pengfei Wan, name: Di Zhang, name: Kun Gai, name: Yujiu Yang, name: Wanli Ouyang

Video generation has achieved significant advances through rectified flow
techniques, but issues like unsmooth motion and misalignment between videos and
prompts persist. In this work, we develop a systematic pipeline that harnesses
human feedback to mitigate these problems and refine the video generation
model. Specifically, we begin by constructing a large-scale human preference
dataset focused on modern video generation models, incorporating pairwise
annotations across multi-dimensions. We then introduce VideoReward, a
multi-dimensional video reward model, and examine how annotations and various
design choices impact its rewarding efficacy. From a unified reinforcement
learning perspective aimed at maximizing reward with KL regularization, we
introduce three alignment algorithms for flow-based models by extending those
from diffusion models. These include two training-time strategies: direct
preference optimization for flow (Flow-DPO) and reward weighted regression for
flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
reward guidance directly to noisy videos. Experimental results indicate that
VideoReward significantly outperforms existing reward models, and Flow-DPO
demonstrates superior performance compared to both Flow-RWR and standard
supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
weights to multiple objectives during inference, meeting personalized video
quality needs. Project page: https://gongyeliu.github.io/videoalign.