Searching Priors Makes Text-to-Video Synthesis Better
Authors: name: Haoran Cheng; name: Liang Peng; name: Linxuan Xia; name: Yuepeng Hu; name: Hengjia Li; name: Qinglin Lu; name: Xiaofei He; name: Boxi Wu
Published: 2024-06-05 12:53:28

Abstract: Significant advancements in video diffusion models have brought substantial
progress to the field of text-to-video (T2V) synthesis. However, existing T2V
synthesis model struggle to accurately generate complex motion dynamics,
leading to a reduction in video realism. One possible solution is to collect
massive data and train the model on it, but this would be extremely expensive.
To alleviate this problem, in this paper, we reformulate the typical T2V
generation process as a search-based generation pipeline. Instead of scaling up
the model training, we employ existing videos as the motion prior database.
Specifically, we divide T2V generation process into two steps: (i) For a given
prompt input, we search existing text-video datasets to find videos with text
labels that closely match the prompt motions. We propose a tailored search
algorithm that emphasizes object motion features. (ii) Retrieved videos are
processed and distilled into motion priors to fine-tune a pre-trained base T2V
model, followed by generating desired videos using input prompt. By utilizing
the priors gleaned from the searched videos, we enhance the realism of the
generated videos' motion. All operations can be finished on a single NVIDIA RTX
4090 GPU. We validate our method against state-of-the-art T2V models across
diverse prompt inputs. The code will be public.