T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through
  Data, Reward, and Conditional Guidance Design
Authors: name: Jiachen Li; name: Qian Long; name: Jian Zheng; name: Xiaofeng Gao; name: Robinson Piramuthu; name: Wenhu Chen; name: William Yang Wang
Published: 2024-10-08 04:30:06

Abstract: In this paper, we focus on enhancing a diffusion-based text-to-video (T2V)
model during the post-training phase by distilling a highly capable consistency
model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2,
introduces a significant advancement by integrating various supervision
signals, including high-quality training data, reward model feedback, and
conditional guidance, into the consistency distillation process. Through
comprehensive ablation studies, we highlight the crucial importance of
tailoring datasets to specific learning objectives and the effectiveness of
learning from diverse reward models for enhancing both the visual quality and
text-video alignment. Additionally, we highlight the vast design space of
conditional guidance strategies, which centers on designing an effective energy
function to augment the teacher ODE solver. We demonstrate the potential of
this approach by extracting motion guidance from the training datasets and
incorporating it into the ODE solver, showcasing its effectiveness in improving
the motion quality of the generated videos with the improved motion-related
metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2
establishes a new state-of-the-art result on VBench, with a Total score of
85.13, surpassing proprietary systems such as Gen-3 and Kling.