SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For video-text alignment, we investigate five metrics. (1) CLIPScore [10] is widely used to evaluate T2V/I generation models. CLIPScore is based on CLIP [29], a vision-language model (VLM) pretrained on large-scale text-image pairs. (2) CLIPScore-ft is based on a CLIP model fine-tuned on the video-text retrieval task using MSR-VTT data. (3) BLIPScore and (4) UMTScore replace CLIP with more advanced VLMs, i.e., BLIP [21] and UMT [22]. BLIP adopts bootstrapping language-image pre-training. UMT is pre-trained large-scale video-text data and we adopt the fine-tuned version on MSR-VTT. Then, we compute BLIPScore in the same way as CLIPScore. For UMT, we utilize the cross-modal decoder's video-text matching result and refer to this metric as UMTScore. We also adopt a Video Large Language Models (LLMs), i.e., Otter [20], to evaluate video-text alignment by formulating it as a Video QA task. Specifically, we first extract the key elements from the text prompt and generate corresponding yes-no questions via the Vicuna model [5]. Then, we feed the generated (or ground-truth) videos and questions into the Video LLM. The average number of questions that receive a positive answer is defined as the alignment score for a specific video-text pair. We name the Video LLM-based metric as (5) Otter-VQA.
For video quality metrics, we investigate FID [11] and FVD [40]. The original FVD adopts the I3D model [3] as a video encoder. We introduce FVD-UMT, an enhanced version of FVD that leverages the UMT vision encoder for video feature extraction. More details of these metrics and our implementations can be found in Appendix D.
Table 2: Correlation between automatic and manual evaluations of video-text alignment, measured by Kendall's  $\tau_{c}$  (left) and Spearman's  $\rho$  (right) coefficients. We also report the average inter-human correlation in the final row as a reference.
Color	Quantity	Camera View	Speed	Motion Direction	Event Order	All
CLIPScore	0.157/0.218	0.147/0.202	0.254/0.345	0.178/0.246	0.141/0.194	0.177/0.248	0.177/0.243
CLIPScore-ft	0.206/0.287	0.250/0.340	0.293/0.402	0.203/0.280	0.254/0.348	0.157/0.221	0.224/0.309
BLIPScore	0.222/0.307	0.207/0.282	0.285/0.394	0.195/0.266	0.223/0.305	0.180/0.250	0.235/0.322
Otter-VQA	0.049/0.070	0.134/0.188	0.027/0.038	0.051/0.073	0.119/0.166	0.146/0.206	0.081/0.114
UMTScore	0.304/0.420	0.394/0.528	0.300/0.415	0.296/0.407	0.356/0.476	0.295/0.406	0.309/0.425
Human	0.547/0.702	0.647/0.784	0.447/0.595	0.539/0.683	0.619/.0.747	0.517/0.680	0.576/0.719
Figure 8: Automatic and human ranking of the T2V models in terms of video-text alignment. Results on specific categories are reported in Appendix F.1.
5.2 Evaluation Setups
Video-Text Alignment Metrics. We use the same generated and reference videos for manual evaluation (one video for a text prompt using one model) and compute the video-text alignment for each video-text pair. Then, we measure the correlation between automatic and manual evaluations, using Kendall's  $\tau_{c}$  coefficient and Spearman's  $\rho$  coefficient. To obtain the correlation under a specific category, we collect the manual and automatic evaluation results of video-text pairs that belong to this category and compute the coefficients.
Video Quality Metrics. Different from the video-text alignment metrics, FID and FVD evaluate the quality of a group of generated videos instead of a single video. To evaluate the fine-grained video quality in specific categories, for each open T2V model, we generate 300 videos for each category under the "major content" aspect, based on FETV prompts. In this case, each prompt can be used multiple times but the generated videos are different due to the randomness in the diffusion process and visual token sampling. We also sample 300 reference videos for each category. This is achieved by using the automatic categorization results of MSR-VTT and WebVid prompts (introduced in Section 3.2) and collecting the corresponding videos. To compute the overall FID and FVD for a T2V model, we sample 1,024 generated and reference videos (the impact of video sample number is analyzed in Appendix F.3). The results are averaged across 4 runs with different random seeds.
5.3 Results and Analysis
Video-Text Alignment Metrics. Table 2 shows the correlation between automatic and manual evaluations. We can see that: (1) Although the widely-used CLIPScore is positively correlated with manual evaluations, the overall correlation degree is very weak compared with inter-human correlations. (2) Fine-tuning CLIP on video-text retrieval and replacing CLIP with BLIP are beneficial, which promote the correlation across almost all categories. However, CLIPScore-ft and BLIPScore are still inconsistent with the human ranking of videos generated from different models, as shown in Figure 8. Therefore, we cannot rely on CLIPScore and BLIPScore to accurately evaluate open T2V models. (3) UMTScore exhibits the strongest correlation with humans when measured by Kendall's  $\tau_{c}$  and Spearman's  $\rho$ . More importantly, it consistently aligns with human judgments in the T2V model rankings. The performance of UMTScore suggests that training VLMs with video data is important and the evaluation of open T2V models can benefit from the development of stronger VLMs. (4) The correlation varies across different "attribute control" categories. It is interesting to see that for UMTScore, the correlation in the challenging categories (for open T2V models), i.e., "Quantity" and "Motions Direction", is higher than in the relatively simple "Color" and "Camera
Figure 9: Video examples and alignment scores measured by different metrics. The left three videos are generated and the rightmost video is the ground-truth. More examples are shown in Appendix F.6.
Figure 10: Automatic and human ranking of T2V models in terms of video quality (averaged static and temporal quality). Results on specific categories are reported in Appendix F.2.
View". We conjecture that this is because, in the simpler categories, the performance of open T2V models is more similar and it is more difficult to distinguish the difference, and vice versa.
Figure 9 presents a case study of video-text alignment scores. We can see that the results measured by CLIPScore and Otter-VQA strongly differ from human evaluation. CLIPScore-ft and BLIPScore correlate better with humans, while still mistakenly assigning lower score to the ground-truth video. In comparison, UMTScore's ranking of video-text alignment is most consistent with humans.
Video Quality Metrics. For FID and FVD, we cannot calculate the correlation coefficients because they are not sample-wise metrics. To investigate their correlation with humans, we focus on the system-wise ranking of open T2V models. As we can see in Figure 10, the FID and the original FVD with I3D video encoder are loosely coupled with the human's perception of the video quality. This result reveals the critical problem of using FID and FVD as a measurement of video quality, which is common in previous works. In comparison, the FVD-UMT metric, which is enhanced with the UMT model as video feature extractor, generates consistent rankings with humans.
In this paper, we propose the FETV benchmark for fine-grained evaluation of open-domain text-to-video generation. Compared with existing benchmarks, FETV is both multi-aspect and temporal-aware, which can provide more specific information on the performance of T2V models. Based on FETV, we conduct a comprehensive manual evaluation of representative T2V models and reveal their pros and cons in different categories of text prompts from different aspects. Furthermore, we extend FETV to diagnose the reliability of automatic T2V metrics, revealing the poor correlation of existing metrics with humans. To address this problem, we develop two automatic metrics using an advanced VLM called UMT, which can produce a consistent ranking of T2V models with humans.
The limitations of this work can be viewed from two perspectives. First, while the proposed UMT-based metrics are more reliable than existing automatic metrics, they still have room for improvement to better align with humans. Second, due to the lack of open-sourced open T2V models, the number of models evaluated in this work is limited. We invite follow-up studies to evaluate their models on our benchmark and encourage more projects to open-source their models.
We thank all the anonymous reviewers for their constructive comments. This work is supported in part by a Huawei Research Grant and National Natural Science Foundation of China (No. 62176002). Xu Sun is the corresponding author of this paper.
[1] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1708-1718. IEEE, 2021.
[2] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[3] J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724-4733. IEEE Computer Society, 2017.
[4] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. CoRR, abs/1808.01340, 2018.
[5] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with  $90\%$  chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
[6] M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In NeurIPS, 2022.
[7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[8] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV (4), volume 12349 of Lecture Notes in Computer Science, pages 214-229. Springer, 2020.
[9] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen. Latent video diffusion models for high-fidelity long video generation. CoRR, abs/2211.13221, 2022.
[10] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP (1), pages 7514-7528. Association for Computational Linguistics, 2021.
[11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, pages 6626-6637, 2017.
[12] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[13] J. Ho, W. Chan, C. Sahara, J. Whang, R. Gao, A. A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models. CoRR, abs/2210.02303, 2022.
[14] J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. In NeurIPS, 2022.
[15] W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. CoRR, abs/2205.15868, 2022.
[16] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017.
[17] L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. CoRR, abs/2303.13439, 2023.
[37] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 2256-2265. JMLR.org, 2015.
[38] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012.
[39] S. Sterling. zeroscope-v2, June 2023.
[40] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. FVD: A new metric for video generation. In DGS@ICLR. OpenReview.net, 2019.
[41] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In NIPS, pages 6306-6315, 2017.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[43] R. Villegas, M. Babaeizadeh, P. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual description. CoRR, abs/2210.02399, 2022.
[44] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang. Modelscope text-to-video technical report. CoRR, abs/2308.06571, 2023.
[45] C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan. GODIVA: generating open-domain videos from natural descriptions. CoRR, abs/2104.14806, 2021.
[46] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan. Niwa: Visual synthesis pre-training for neural visual world creation. In ECCV (16), volume 13676 of Lecture Notes in Computer Science, pages 720-736. Springer, 2022.
[47] J. Z. Wu, Y. Ge, X. Wang, W. Lei, Y. Gu, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. CoRR, abs/2212.11565, 2022.
[48] J. Xu, T. Mei, T. Yao, and Y. Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296. IEEE Computer Society, 2016.
[49] R. Yang, P. Srivastava, and S. Mandt. Diffusion probabilistic modeling for video generation. CoRR, abs/2203.09481, 2022.
[50] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang, J. Baldridge, and Y. Wu. Scaling autoregressive models for content-rich text-to-image generation. Trans. Mach. Learn. Res., 2022, 2022.
[51] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng. Magicvideo: Efficient video generation with latent diffusion models. CoRR, abs/2211.11018, 2022.
A Overview of Appendix
B: More Details of FETV.
C: Text-to-Video Generation Model Details.
D: More Details of Automatic Evaluation Setups.
E: More Details of Manual Evaluation Setups.
F: More Experimental Results.
G: Automatic Categorization Rules.
H: Licensing, Hosting and Maintenance Plan.
I: Datasheet.
B More Details of FETV
B.1 Category Descriptions
In Figure 2 of the main paper, we illustrate the descriptions and examples of the temporal categories. In this appendix, Figure 11 presents the descriptions and examples of the spatial categories under the "major content" and "attribute control" aspects. Table 3 presents the descriptions and examples of different prompt complexity levels.
B.2 Data Distribution
Figure 12 compares FETV with MSR-VTT and WebVid in terms of their data distribution over categories. To obtain the distribution of MSR-VTT and WebVid, we randomly sample 100 data from each dataset and manually annotate the categories as we construct FETV. As we can see, FETV exhibits a more uniform distribution over the categories, while MSR-VTT and WebVid are more biased toward certain categories. Particularly, more than  $60\%$  of the MSR-VTT and WebVid prompts do not involve the six attributes introduced in FETV. Consequently, the ability to control these categories, especially the challenging ones, cannot be reflected by the performance on MSR-VTT and WebVid (note that these two datasets lack categorization of the attributes).
B.3 Data Examples
Table 4 shows FETV prompt examples with corresponding categorizations, which cover all 22 categories under the three aspects.
C Text-to-Video Generation Model Details
C.1 CogVideo
Model Architecture. CogVideo [15] generates videos as discrete visual tokens, encoded by the VQVAE[41], using the Transformer model [42]. The generation process consists of two stages. In the first stage, a Transformer sequentially generates some keyframes based on the text prompts. In the second stage, another Transformer generates the remaining frames by interpolating the generated frames in a recursive way. In both stages, CogVideo can generate videos according to a specified frame rate. CogVideo is initialized from the pre-trained text-to-image model CogView2 [6] and inserts a temporal channel attention to model the dynamics in videos. In total, the Transformer of both stages has 9.4 billion parameters, among which 6 billion parameters are from CogView2.
Training Setups. CogVideo is trained on a private dataset of 5.4 million captioned videos with  $160 \times 160$  spatial resolution. The first-stage Transformer is trained with a 1 fps frame rate and the second-stage Transformer is trained with 2,4,8 fps frame rates.
Implementation Details. For inference of CogVideo, we adopt the official implementation with default hyper-parameters setups. We use the Transformer models of both stages and generate videos with 33 frames (a duration of 4 seconds), with a spatial resolution of  $480 \times 480$  (upsampled by CogView2). The inference of CogVideo consumes approximately 24 GB of GPU memory. We use two types of GPU, i.e., Nvidia A40 and TITAN RTX. The inference of all four T2V models is run on a single GPU. The codes are based on the Pytorch framework.
(a) Spatial categories under the "major content" aspect.
(b) Spatial categories under the "attribute control" aspect.
Figure 11: Descriptions and examples of the spatial categories.
(a) Data distribution of 200 random samples from MSR-VTT and WebVid.
(b) Data distribution of FETV.
Figure 12: Percentile distribution of data across categories. "None" denotes the data samples that do not belong to any of the categories under certain aspects. Note that the distributions do not sum to one because each data sample can belong to multiple categories.
Table 3: Descriptions and examples of different "prompt complexity" categories.
Category	Description	Example Prompts
Simple	Prompts that involve 0 ~ 4 non-stop words.	Prompt1: the cars drove fast
Prompt2: people are dancing
Prompt3: a dog is sneezing
Medium	Prompts that involve 5 ~ 8 non-stop words.	Prompt1: A man is typing on a wireless keyboard . close up.
Prompt2: flying counter clockwise around a large yacht
Prompt3: aerial sunrise shot over a town, flying forwards
Complex	Prompts that involve more than 8 non-stop words.	Prompt1: Tilt down slow motion wide tracking shot of man walking along stony ground towards clouds
Prompt2: in a cookery program the preparation of vegetable soup is shown cutting leaves and add pinch of salt to water and boiling
Prompt3: a corn field moves quickly past the camera there is a shot of a city and then of a double-decker passenger train
C.2 Text2Video-Zero
Model Architecture. Text2Video-Zero [17] adapts text-to-image generation models for zero-shot text-to-video generation without any training on video data. Specifically, it introduces three techniques: (1) injecting motion dynamics into the first frame's latent code to obtain a sequence of dynamic frames; (2) replacing each frame's self-attention with cross-attention to the first frame to enhance temporal coherence; (3) a background smoothing strategy to enhance the consistency of background across frames.
Implementation Details. Text2Video-Zero is compatible with a wide range of diffusion-based text-to-image models. In this work, we utilize the dreamlike-photoreal-2.0 version of Stable Diffusion, following the default setup of Text2Video-Zero's official implementation. We generate videos of 2
Table 4: Prompts and categorizations of FETV data examples. "Fluid" denotes fluid motions; "Kinetic" denotes kinetic motions; "Light" denotes light change; "Buildings" denotes buildings & infrastructures; "Food" denotes food & beverage; "Scenery" denotes scenery & natural objects.
Prompt	Major Content	Attribute Control	Prompt Complexity
Spatial	Temporal	Spatial	Temporal
a person is cooking	People	Actions	None	None	Simple
rabbits are running around	Animals	Actions	None	None	Simple
someone folds paper	Artifacts	Actions	None	None	Simple
A mountain stream	Scenery	Fluid	None	None	Simple
a fire is burning	Scenery	Light	None	None	Simple
sea side road	Scenery;Buildings	Fluid	None	None	Simple
someone driving an orange sports car	People;Vehicles	Actions;Kinetic	Color	None	Medium
girl in pink dress fashion model walking in ramp	People;Artifacts	Actions;Kinetic	Color	None	Medium
Rows of young green sunflower plants in field in sunset, agriculture in spring	Plants	Light	Color	None	Complex
three boys are eating burger	People;Food	Actions	Quantity	None	Simple
four airplanes flying	People;Vehicles	Actions;Kinetic	Quantity	None	Simple
a first person view of a man driving a red formula one car	People;Vehicles	Actions;Kinetic	Color;Camera	None	Complex
overhead view as pingpong players compete on the table	People;Artifacts	Actions	Camera	None	Medium
Toasting with beer, close up view.	Food	Fluid	Camera	None	Simple
Bees flying, slow motion shot	Animals	Actions;Kinetic	None	Speed	Medium
the cars drove fast	Vehicles	Kinetic	None	Speed	Simple
Time lapse moving clouds with blue sky	Scenery	Fluid	Color	Speed	Medium
Flying counter clockwise around a large yacht	Vehicles	Kinetic	None	Direction	Medium
A wave approaching and crashing on to a big rock on the shore.	Scenery	Fluid	None	Direction	Medium
a man walks away from an suv	People;Vehicles	Actions;Kinetic	None	Direction	Simple
Abstract circular light pattern, artistic blurred background, camera moving from left to right	Illustrations	Kinetic;Light	None	Direction	Complex
an old man shakes hands with another man and then they hug each other	People	Actions	None	Order	Medium
a guy places a pan into tin foil and then places the pan into a water bath	People;Artifacts	Actions;Fluid	None	Order	Complex
a car first drives on a mountain road and then on a highway along the lake	Vehicles;Scenery; Buildings	Kinetic	None	Order	Medium
seconds at 8 fps, with a spatial resolution of  $512 \times 512$ . The inference of Text2Video-Zero consumes approximately 24 GB of GPU memory.
C.3 ModelScopeT2V
Model Architecture. ModelScopeT2V [44] extends the T2I Stable Diffusion model with temporal convolution and attention blocks, which enables video modelling. Then, the model is trained using both video-text and image-text datasets, which mitigates catastrophic forgetting of the T2I generation ability.
Training Setups. ModelScopeT2V is trained on both video-text pairs (WebVid-10M [1]) and image-text pairs (LAION2B-en [33]). During training, 1/8 of GPUs are allocated to images and the remaining 7/8 GPUs are used for videos.
Implementation Details. We implement ModelScopeT2V using the code and model checkpoints released in ModelScope. We generate videos of 2 seconds at 8fps, with a spatial resolution of  $256 \times 256$ . The inference of ModelScopeT2V consumes approximately 16 GB of GPU memory.
C.4 ZeroScope
Model Architecture and Training Setups. ZeroScope [39] is a watermark-free video generation model optimized for producing 16:9 compositions. It is trained from the weights of ModelScopeT2V using 9,923 clips and 29,769 tagged frames at 24 frames, 576x320 resolution.
Implementation Details. We adopt the ZeroScope-v2-576w model, following the official implementation in HuggingFace. The generated videos are 3 seconds at 8fps, with a spatial resolution of  $576 \times 320$ . The inference of ZeroScope-v2-576w consumes approximately 20 GB of GPU memory.
D More Details of Automatic Evaluation Setups
D.1 Video-Text Alignment Metrics