LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
Authors: name: Yibin Wang; name: Zhiyu Tan; name: Junyan Wang; name: Xiaomeng Yang; name: Cheng Jin; name: Hao Li
Published: 2024-12-06 07:16:14

Abstract: Recent advancements in text-to-video (T2V) generative models have shown
impressive capabilities. However, these models are still inadequate in aligning
synthesized videos with human preferences (e.g., accurately reflecting text
descriptions), which is particularly difficult to address, as human preferences
are inherently subjective and challenging to formalize as objective functions.
Therefore, this paper proposes LiFT, a novel fine-tuning method leveraging
human feedback for T2V model alignment. Specifically, we first construct a
Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k
human annotations, each including a score and its corresponding rationale.
Based on this, we train a reward model LiFT-Critic to learn reward function
effectively, which serves as a proxy for human judgment, measuring the
alignment between given videos and human expectations. Lastly, we leverage the
learned reward function to align the T2V model by maximizing the
reward-weighted likelihood. As a case study, we apply our pipeline to
CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B
across all 16 metrics, highlighting the potential of human feedback in
improving the alignment and quality of synthesized videos.