I4VGen: Image as Stepping Stone for Text-to-Video Generation
Authors: name: Xiefan Guo; name: Jinlin Liu; name: Miaomiao Cui; name: Di Huang
Published: 2024-06-04 11:48:44

Abstract: Text-to-video generation has lagged behind text-to-image synthesis in quality
and diversity due to the complexity of spatio-temporal modeling and limited
video-text datasets. This paper presents I4VGen, a training-free and
plug-and-play video diffusion inference framework, which enhances text-to-video
generation by leveraging robust image techniques. Specifically, following
text-to-image-to-video, I4VGen decomposes the text-to-video generation into two
stages: anchor image synthesis and anchor image-guided video synthesis.
Correspondingly, a well-designed generation-selection pipeline is employed to
achieve visually-realistic and semantically-faithful anchor image, and an
innovative Noise-Invariant Video Score Distillation Sampling is incorporated to
animate the image to a dynamic video, followed by a video regeneration process
to refine the video. This inference strategy effectively mitigates the
prevalent issue of non-zero terminal signal-to-noise ratio. Extensive
evaluations show that I4VGen not only produces videos with higher visual
realism and textual fidelity but also integrates seamlessly into existing
image-to-video diffusion models, thereby improving overall video quality.