SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanxin Liu $^{\S}$ , Lei Li $^{\S}$ , Shuhuai Ren $^{\S}$ , Rundong Gao $^{\mathbb{Q}}$ , Shicheng Li $^{\S}$ , Sishuo Chen $^{\mathbb{Q}}$ , Xu Sun $^{\S}$ , Lu Hou $^{\ddagger}$
$\S$  National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University
$\mathbb{I}$  Center for Data Science, Peking University  $\ddagger$  Huawei Noah's Ark Lab {liuyuanxin, shuhuai_ren, gaord20}@stu.pku.edu.cn nlp.lilei@gmail.com {lisc99, chensishuo, xusun} @pku.edu.cn houlu3@huawei.com
Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/11yx97/FETV.
Open-domain text-to-video (T2V) generation, which involves synthesizing videos in response to free-form textual prompts, has experienced significant advancements in recent years. Contemporary T2V models have demonstrated impressive visual quality and alignment with prompts. However, this progress is mainly supported by the qualitative cases of generated videos. To accurately assess the current state of T2V generation, enable model comparisons, and guide future research, it is essential to develop robust quantitative evaluation systems
However, existing quantitative evaluations for T2V models present two major challenges: (1) Limited scope of performance measurements. Current studies typically report overall results on entire test sets, providing only a coarse view of model performance. A more nuanced, fine-grained evaluation is necessary to gain a comprehensive understanding of model capabilities. While some benchmarks categorize text prompts based on content or challenge type [31, 50], they often neglect the temporal
Table 1: Comparison of text-to-image/video generation benchmarks.  $\times$  and  $\checkmark$  denote whether the benchmark contains the corresponding aspect (column) of categorization or whether it is open-domain. FETV is open-domain, multi-aspect and temporal-aware, compared with other benchmarks.
Task Type	Benchmark	Open Domain	Major Content	Attribute Control	Prompt Complexity
spatial	temporal	spatial	temporal
Text2Image	DrawBench	✓	✗	✗	✓	✗	✗
PartiPrompts	✓	✓	✗	✓	✗	✓
Text2Video	UCF-101	✗	✗	✓	✗	✗	✗
Kinetics	✗	✗	✓	✗	✗	✗
MSR-VTT	✓	✓	✗	✗	✗	✗
Make-a-Video-Eval	✓	✓	✗	✗	✗	✗
FETV (Ours)	✓	✓	✓	✓	✓	✓
aspects of video generation. (2) Lack of reliable automatic evaluation metrics. In text-to-image generation, it has been found that the existing automatic evaluation metrics are inconsistent with human judgment [6, 27, 28]. However, in T2V generation, this problem is still under-explored.
To address these issues, we introduce a benchmark called FETV for the Fine-grained Evaluation of Text-to-Video generation, which offers a comprehensive categorization system for T2V evaluation. FETV includes a diverse set of text prompts, categorized based on three orthogonal aspects: major content, attribute control, and prompt complexity (see Figure 1). We also propose several temporal categories specifically designed for video generation, encompassing both spatial and temporal content. As shown in Table 1, compared with existing benchmarks, FETV highlights the most comprehensive categorization for fine-grained evaluation of T2V generation. To efficiently categorize the text prompts, we employ a two-step annotation process involving an automatic assignment of category labels followed by manual review. On this basis, we collect 619 categorized text prompts from existing open-domain text-video datasets [48, 1] and manually-written unusual prompts.
Using FETV, we perform comprehensive manual evaluations for four representative T2V models, enabling fine-grained analysis of their strengths and weaknesses across different prompt categories. The results highlight the weaknesses of existing T2V models in terms of (1) poor video quality for action and kinetic motion related videos and (2) inability to control quantity, motion direction and event order (see Section 3.1 for detailed definition of these categories). Furthermore, FETV can serve as a testbed for assessing the reliability of automatic evaluation metrics through correlation analysis with manual evaluations. Our experiments reveal that the widely used automatic metrics of video quality (FID [11] and FVD [40]) and video-text alignment (CLIPScore [10]) correlate poorly with human evaluation. In response to this problem, we explore several solutions to improve CLIPScore and FVD. Based on UMT [22], an advanced vision-language model (VLM), we develop FVD-UMT for video quality and UMTScore for video-text alignment, which exhibit much higher correlation with humans than existing metrics.
The contributions of this work are as follows: (1) We propose the FETV for fine-grained evaluation of open-domain T2V generation. Compared with existing benchmarks, FETV provides a more comprehensive view of T2V models' capabilities from multiple aspects. (2) Based on FETV, we conduct a comprehensive manual evaluation of four representative open T2V models and reveal their pros and cons from different aspects. (3) We extend FETV as a testbed to evaluate the reliability of automatic T2V metrics and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics.
Benchmarks for Text-to-Video Generation. Most existing studies on T2V generation adopt datasets from other related tasks to evaluate their models. UCF-101 [38] and Kinetics-400/600/700 [16, 4, 36] are initially proposed for the task of human action recognition, which cannot cover the diversity of open-domain video generation. MSR-VTT [48], which is commonly used for video captioning and video-text retrieval, consists of text-video pairs covering 20 categories with diverse video contents. However, its categorization only considers the topic of videos (e.g., "music", "sports" and "news") while ignoring other important aspects in text-to-video generation, e.g., the temporal
(a) Categorization of a Single Data
(c) Categorization of Test Set for Fine-Grained Evaluation
(b) Summary of All Categories
Figure 1: Illustration of our multi-aspect categorization (a-b), based on which we can realize fine-grained evaluation (c). Details of each category are shown in Figure 2 and Appendix B.1.
Attribute Control
color; q
speed; motion direction;
Prompt Complexity
simple; medium; complex
contents and the attributes a text prompt aims to control. In text-to-image generation, DrawBench [31] and PartiPrompts [50] introduce another aspect of categorization concerning the type of challenge a prompt involves. However, their prompts lack descriptions of temporal content, which are not suitable for T2V generation. Make-a-Video-Eval [34] contains 300 text prompts specially composed for T2V generation. However, their categorization (i.e., animals, fantasy, people, nature and scenes, food and beverage) only considers the spatial content.
Automatic Metrics for Text-to-Video Generation. There are two types of automatic open T2V metrics, which evaluate the video quality and video-text alignment, respectively. For video quality, the widely used metrics are the Inception Score (IS) [32], Frechet Inception Distance (FID) [11], and Frechet Video Distance (FVD) [40]. Based on a classification model, IS assesses whether every single generated video contains identifiable objects and whether all the videos are diverse in the generated objects. FID and FVD compute the distribution similarity between real and generated videos in the latent space of a deep neural network. In terms of video-text alignment, existing works on open T2V mainly adopt the CLIPScore [10]. CLIPScore is initially proposed to measure the similarity between machine-generated captions and the given image, using the pre-trained CLIP model [29]. In the task of text-to-image generation, it has been found that the automatic image quality and image-text alignment metrics are inconsistent with human judgment [6, 27, 28]. When it comes to open T2V, this problem remains under-explored.
Models for Text-to-Video Generation. Existing text-to-video generation models can be divided into the Transformer-based models [45, 46, 15, 43] and the Diffusion-based models [14, 13, 34, 51, 17, 25, 2, 9, 44]. The former first converts the videos into discrete visual tokens and then trains the Transformer model to generate these visual tokens. The latter generates videos using the Diffusion Models [37, 12, 49], either on the original pixel space or a latent space. In this paper, We evaluate three models that are open-sourced up to the time of writing this paper, namely CogVideo [15], Text2Video-zero [17] and ModelScopeT2V [44], as well as a more recent model ZeroScope [39]. Note that text-driven video editing [26, 47], where the input is a text prompt and a source video, may also be referred to as text-to-video generation. In this paper, we focus on the typical kind of text-to-video generation, where the input condition is only a piece of text prompt.
3 FETV Benchmark
FETV is a dataset of text prompts divided into different categories from multiple aspects for fine-grained evaluation of T2V generation. In this section, we will introduce the multi-aspect categorization, how to collect the prompts and category labels and the application scope of FETV.
(a) Temporal categories under the "major content" aspect.
(b) Temporal categories under the "attribute control" aspect.
Figure 2: Descriptions and examples of the temporal categories.
3.1 Multi-Aspect Categorization
As shown in Figure 1, each prompt is categorized based on three aspects, namely the major content it describes, the attribute it aims to control in the generation of videos, and its complexity. The "major content" aspect is further divided into spatial and temporal categories, the former focus on the type of objects described in the prompt, while the latter focus on the type of temporal content. Similarly, the "attribute control" aspect is comprised of both spatial attributes and temporal attributes. The "prompt complexity" aspect has three levels of complexity, namely "simple", "medium", and "complex", depending on the number of non-stop words contained in a prompt. With the multi-aspect categorization of every single prompt, the entire FETV benchmark can be divided into different subsets (Figure 1 (c)), therefore enabling fine-grained evaluation. Figure 2 presents the descriptions and corresponding examples of the temporal categories. The complete description of all categories can be found in Appendix B.1.
3.2 Data Collection
Prompt Sources. The prompts of FETV come from two kinds of sources. (1) we reuse the texts from existing datasets of open-domain text-video pairs, which guarantees the diversity of prompts. Specifically, we use the MSR-VTT [48] test set and WebVid [1] validation set, which contain 59,800 and 4,999 text-video pairs, respectively. (2) we manually write prompts describing scenarios that are unusual in the real world, which cannot be found in existing text-video datasets. These "unusual prompts" are inspired by the "Imagination" category in PartiPrompts [50] and "Conflicting" category in DrawBench [31], which aims to test the generalization ability of the generation models. To write the unusual prompts, we first define several types of unusual cases, namely "object-action", "object-object", "object-color", "object-quantity", "object-direction", "object-speed" and "event order". For example, the prompt "A cat is driving a car." belongs to the case of unusual "object-action". Then, for each case, we enumerate several specific objects and attributes and write the prompts.
Prompt Categorization and Selection For the prompts from both sources, we assign category labels to them based on our multi-aspect categorization structure. This is achieved in two steps: In the first step, we define a list of keyphrases, WordNet synsets and some hand-crafted matching patterns for each category and automatically categorize the prompts. For example, to determine
Figure 3: Data distribution over categories under the "major content" (upper) and "attribute control" (lower) aspects.
Figure 4: Data distribution over categories under the "prompt complexity" aspect.
whether a prompt belongs to the "animals" category, we iterate over the words in the prompt and see whether any of them belong to the animal-related WordNet synset. Details of the automatic matching rules for each category are presented in Appendix G. In the second step, we manually select prompts for each category and make some revisions to the category labels to ensure that they are correct. Specifically, we enumerate combinations of  $(c_i, c_j)$ , where  $c_i$  belongs to the six "attribute control" categories and  $c_j$  belongs to the four "temporal major content" categories. For each combination, we select roughly 20 prompts that belong to both  $c_i$  and  $c_j$ , according to the first-step categorization, and then revise the incorrect category labels. For the text prompts from existing datasets, we also revise the ones that do not align well with the reference video.
3.3 Dataset Statistics and Data Format
We collect a total of 619 prompts with corresponding categorizations, among which 541 prompts are from existing datasets and 78 unusual prompts are written by us. Such data scale is comparable with existing text-to-image/video benchmarks, e.g., DrawBench (200 prompts), PartiPrompts (1,600 prompts) and Make-a-Video-Eval (300 prompts), which are mainly proposed for manual evaluation. The data distributions over categories under different aspects are illustrated in Figure 3 and Figure 4, which show that FETV contains enough data to cover the diverse range of categories. We also compare FETV with MSR-VTT and WebVid in terms of the data distribution over categories, which can be found in Appendix B.2.
Each data sample in FETV consists of three elements: the prompt  $t$ , the reference video  $v^{\mathrm{ref}}$  (not available for the unusual prompts) and the categorization labels  $\{\mathcal{C}^a\}_{a \in \mathcal{A}}$ .  $\mathcal{A}$  is the collection of the three aspects,  $\mathcal{C}^a$  is the categorization labels of  $t$  under aspect  $a$ . Taking the prompt in Figure 1 as an example, when the aspect  $a$  is "attribute control",  $\mathcal{C}^a = \{"speed"\}$ . Note that except for the complexity aspect, we allow each prompt to be classified into multiple categories under the same aspect. This setting is more realistic compared with existing benchmarks that only classify each prompt into a single category. More examples of FETV data can be found in Appendix B.3.
3.4 Application Scope
Theoretically, FETV can be used for both manual and automatic evaluation. However, considering that existing automatic T2V metrics correlate poorly with humans (see Section 5), we mainly rely on manual evaluation to obtain more accurate results. In addition to evaluating T2V models, FETV can also be extended to diagnose automatic metrics, using the manual evaluation as a reference.
4 Manual Evaluation of Open T2V Models
4.1 Models
We select four representative T2V models, which are open-sourced, for evaluation. We briefly describe them as follows and defer the details to Appendix C.
CogVideo [15] is based on the Transformer architecture [42]. It first transforms the video frames into discrete visual tokens using VQVAE [41] and then utilizes two Transformer models to hierarchically generate the visual tokens from a given text prompt.
Text2Video-zero [17] is based on the Diffusion Probabilistic Model [37]. It generates videos by enhancing the latent codes of a text-to-image Stable Diffusion model [30] with motion dynamics, without training the model on any video data.
ModelScopeT2V [44] is another diffusion-based model. It extends the T2I Stable Diffusion model with temporal convolution and attention blocks and conducts training using both image-text and video-text datasets.
ZeroScope [39] is a watermark-free T2V generation model trained from the weights of ModelScopeT2V. It is optimized for 16:9 compositions.
4.2 Evaluation Setups
Evaluation Criteria. We evaluate the generated videos mainly from four perspectives: Static quality focuses on the visual quality of single video frames. Temporal quality focuses on the temporal coherence of video frames. Overall alignment measures the overall alignment between a video and the given text prompt. Fine-grained alignment<sup>2</sup> measures the video-text alignment regarding specific attributes. For the first three perspectives, we score the videos based on a 1-5 Likert-scale judgement. For fine-grained alignment, we adopt 1-3 Likert-scale judgement. To facilitate the consistency between human evaluators, we carefully design the wording of questions and descriptions of each rating level. The agreement between our human evaluators is high, with Krippendorff's  $\alpha$  [19] of 0.711, 0.770, 0.638 and 0.653 in the above four perspectives. In Appendix E, we summarize the details of our instruction document for manual evaluation and the annotation interface.
Implementation Details. We generate three videos for each prompt using the three models respectively. For each generated video and each of the above four perspectives, we collect three rating scores from graduate students in our laboratory and report the average rating. For the generated videos, we obtain 619 samples  $\times 3$  models  $\times 3$  perspectives  $\times 3$  humans  $= 16,713$  ratings for the first three perspectives and 5,184 ratings of fine-grained alignment perspective (note that not every prompt involves an attribute for evaluating fine-grained alignment). We also evaluate the 541 reference videos, which produce 6,219 ratings for the four perspectives. In total, we collect 28,116 ratings in our manual evaluation. To obtain the performance under a specific category, we report the average rating of all videos whose prompts belong to this category.
4.3 Results and Analysis
In this section, we present and analyse the results from a quantitative perspective (some qualitative analyses are shown in Appendix F.5). Specifically, we are interested in the following questions: (1) how video quality varies when generating different major contents, (2) how video-text alignment varies when controlling different attributes or facing different prompt complexity. We also investigate (3) how well the models generalize to unusual prompts in Appendix F.4.
The results of video quality are presented in Figure 5. We can observe that: (1) In terms of the spatial categories, the generated videos containing "people" and "animals" have poor quality, while the videos containing "plants" and "scenery & natural objects" are of good quality. This is because videos of the former two categories involve many high-frequency details (e.g., human faces and fingers) which are more difficult to learn than the low-frequency information in the latter two categories (e.g., water and clouds). (2) When it comes to the temporal categories, the videos about "actions" and "kinetic motions" are of worse quality, since the corresponding temporal content is of higher frequency. Although there is a correlation between some spatial and temporal categories
<sup>1</sup>We use the dreamlike-photoreal-2.0 version, following the default setup of Text2Video-zero.
In this context, we reuse the word "fine-grained" to describe alignment between a single video-text pair, while in other contexts it refers to the evaluation of T2V models on specific categories of prompts.
(a) Results across spatial major contents.
(b) Results across temporal major contents.
(a) Results across attributes to control.
Figure 6: Manual evaluation of video-text alignment.
Figure 5: Manual evaluation of static and temporal video quality.
(b) Results across complexity.
(e.g., “actions” is related to “people”), our temporal categorization provides a more direct view of the model performance from the temporal content perspective. (3) None of the four models achieve comparable video quality with real videos, especially in the aforementioned poor-quality categories. (4) Text2Video-zero performs the best in static quality while lagging far behind in temporal quality. This is because Text2Video-zero inherits the image generation power of Stable Diffusion, but the absence of video training prevents it from generating temporally coherent content. (5) ZeroScope, which is trained from ModelScopeT2V, is comparable with ModelScopeT2V in temporal quality
(a) Static Quality
(b) Temporal Quality
(c) Overall Alignment
Figure 7: Leaderboard on FETV benchmark based on manual evaluation.
while performing worse in static quality. We conjecture that this is because the ZeroScope is primarily optimized for the 16:9 composition and watermark-free videos, instead of for better visual quality and video-text alignment.
The results of video-text alignment are shown in Figure 6. We can see that: (1) Existing T2V models can already control "color" and "camera view" in most cases. (2) All four model struggle to accurately control "quantity", "motion direction" and "event order". However, ModelScopeT2V exhibits some preliminary ability to control "speed" and "motion direction", which is not observed in CogVideo and Text2Video-zero. (3) ZeroScope slightly underperforms the original ModelScopeT2V in video-text alignment. We attribute this to the same underlying cause as explained for the video quality performance. (4) The relative performance among different attributes and T2V models is similar between fine-grained and overall alignment, with a few exceptions. This is because the overall alignment may inevitably be affected by other attributes. Therefore, the fine-grained alignment is more accurate when evaluating controllability over specific attributes, but the overall alignment is an acceptable approximation. (5) While video-text alignment is slightly lower for the "complex" prompts, we cannot observe a clear correlation between prompt complexity and video-text alignment.
In addition to the fine-grained evaluating results, we also summarize the leaderboard on the entire FETV benchmark in Figure 7.
5 Diagnosis of Automatic Evaluation Metrics
In this section, we perform automatic evaluations on FETV and diagnose their correlation with manual evaluations.