the current state of t2i models cannot even determine where is what and what place to what and how that should move in which direction compare to the rest. it cannot generate an animated image or video with kinetically animated background, for example: something running in the middle of a street. here's some more text for you, bastert: ```Experiments
Implementation Details

In our experimental setup, we generate $L=16$ frames with a resolution $512$ for each video. We trained the Dual-Stream Diffusion Net using a subset (comprising 5M videos) from the WebVid-10M [1] and HD-VILA-100M [20] datasets. Video clips within the dataset are first sampled at a stride of 4, then resized and centrally cropped to a resolution of $256\times 256$. For the content basic unit, we employ stable diffusion v1.5 pre-trained weights which remain frozen throughout the training procedure. The content incremental unit does not commence training from scratch, but rather utilizes an existing model [15] as pre-trained weights, followed by fine-tuning on our training data. For the motion unit, we initialize our Personalized Motion Generation Stream with the weights of LDM [1], which were pre-trained on Laion-5B [1]. During inference, it takes approximately 35 seconds to sample a single video using one NVIDIA RTX 4090 GPU.

Figure 4: Details of Motion Decomposer and Motion Combiner.

Figure 3: Dual-stream transformation block.
Comparison with Baselines

We compare our method with two publicly available baseline: 1) CogVideo [14]: a Text-to-Video model trained on a dataset of 5.4 million captioned videos, and is capable of generating videos directly from text prompts in a zero-shot manner. 2) Text2Video-Zero [13]: also a Text-to-Video generation method based on the Stable Diffusion model. Since our method is a text-to-video method we compare with Text2Video-Zero in pure text-guided video synthesis settings. Owing to space constraints, we present a quantitative comparison of our method with the two aforementioned approaches. However, for qualitative results, we limit our comparative analysis to the superior performing text2video-zero method for a more focused and meaningful evaluation.

Quantitative ComparisonWe evaluate our method in relation to baseline models by employing automated metrics, detailing frame consistency and textual alignment outcomes in the accompanying Table. 1. For assessing frame consistency, we compute CLIP [13] image embeddings on all frames within the output videos, reporting the average cosine similarity across all pairings of video frames. To evaluate textual alignment, we calculate the average CLIP score between all frames in the output videos and their corresponding prompts. Our findings reveal that the videos generated via our method surpass publicly accessible alternatives such as CogVideo [14] and Text2Video-Zero [13], particularly with regard to frame consistency and text alignment. This suggests that our method offers a more robust and coherent approach to video generation from textual prompts.

Qualitative ComparisonWhen compared to text2video-zero, our method demonstrates superior consistency in both content and motion across generated videos, as shown in Fig. 5. As illustrated in the first row, observable body swings are evident as the panda walks, while in the second row, we see the limbs swing as the person runs, accompanied by gradual changes in the background. In the third row, the lower limbs of the horse are seen swinging as it gallops, set against a dynamic background. Furthermore, our method outperforms the other approach in terms of content quality and its conformity with the text. For instance, the generated pandas in our model appear more realistically rendered, the snow in the second row exhibits imprints, and the street in the third row is more logically constructed. We further include a fourth row as a comparative example in a less favourable environment - rain conditions. Here, the content generated by the other method appears unrealistic, whereas our method not only captures the essence of a rainy day more effectively but also establishes the logical connection between rain and umbrellas, thereby enhancing the realism and context-appropriateness of generated videos.

Beyond this, we further show the qualitative results of our method on video diversity generation, as shown in Fig. 6. Using "a cat is walking on the grass" as the text input, we can see various actions such as a cat walking forward, left, and right. Impressively, the generated videos also exhibit diversity in aspects like fur color, body shape, and pose, thereby encompassing a rich variety in content. Concurrently, the generated video preserves high continuity. As demonstrated in the first row, the flower at the lower right gradually comes into view, while in the second row, subtle changes in the cat's shadow can be discerned. In the third row, a figure in the background is progressively moving, further enhancing the sense of dynamic realism. Furthermore, we acknowledge a minor failure case depicted at the end of the third row, where the color of the cat appears slightly altered. This issue primarily stems from the generated background inadvertently influencing the content creation process itself. However, as evident from the comprehensive results showcased, such instances are rare, thereby attesting to the robustness and reliability of our proposed method in text-to-video generation.

\begin{table}
\begin{tabular}{l c c} \hline \hline Methods & Frame Consistency & Textual Alignment \ \hline CogVideo & 88.32 & 22.02 \ Text2Video-Zero & 90.21 & 29.56 \ Ours & 92.13 & 32.23 \ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of CLIP score metric with baselines.

Figure 5: Qualitative comparison between Text2Video-Zero [13] (frames 1-4 in each row) and our method (frames 5-8 in each row). Please see the videos in the website.

Figure 6: The diversity of our method qualitative results. Prompt: a cat is walking on the grass.```