FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
Authors: Yuanxin Liu
Lei Li
Shuhuai Ren
Rundong Gao
Shicheng Li
Sishuo Chen
Xu Sun
Lu Hou
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: text-to-video generation
fine-grained evaluation
multi-aspect benchmark
TL;DR: This paper introduces a multi-aspect benchmark for fine-grained evaluation of open-domain text-to-video generation models and metrics.

Abstract: Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose **FETV**, a benchmark for **F**ine-grained **E**valuation of **T**ext-to-**V**ideo generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. 
Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.

FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
Yuanxin Liu Lei Li', Shuhuai Ren Rundong Gao", Shicheng Li Sishuo Chen!, Xu Sun Lu Hou*
National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
Center for Data Science, Peking University Huawei Noah's Ark Lab (liuyuanxin, shuhuai_ren, gaord20)@stu. pku.peuuu edu. cn {lisc99, chensishuo, musun)@pkin edu.cn houlu3Ghuawei. com
com
Abstract
Recently, open-domain text-to-video (T2V) generation models have made remark- able progress. However, the promising results mainly shown by the qualitative cases of generated videos, while the quantitative evaluation T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories text prompts. Although some bench- marks have categorized the prompts, their categorization either only focuses a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine- greeeggaaned Evaluation of Text-to- Video generation. FETV is multi-aspect categorizing the prompts based on three orthogonal aspects: the major content, the attributes control and the prompt complexity. FETV also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as testbed to evaluate the re- liability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIIPSore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://git trutt.uu..co www.tttu...mmmmleeeeeee www.o//wmm/////////ccccc 9F//T
Introduction
Open-domain text-to-video (T2V) generation, which involves synthesizing videos in response to free-form textual prompts, has experienced significant advancements in recent years. Contemporary T2V models have demonstrated impressive visual quality and alignment with prompts. However, this progress is mainly supported by the qualitative cases of generated videos. To accurately assess the current state of T2V generation, enable model comparisons, and guide future research, it essential to develop robust quantitative evaluation systems
However, existing quantitative evaluations for T2V models present two major challenges: (1) Limited scope of performance measurements. Current studies typically report overall results entire test sets, providing only a coarse view of model performance. A more nuanced, finennrained evaluation necessary to gain comprehensive understanding of model capabilities. While some benchmarks categorize text prompts based on content or challenge type [1 (5),, they often neglect the temporal
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table I: Comparison of text-to-image/videe generation benchmarks. x and dennte whether the benchmark contains the corresponding aspect (column) of categorization or whether it is open-domant FETV is open-domain, multi-aspect and temporal-awan compared with other benchmarks.
Open Domain
Major Content
Attribute Control
Prompt Complexity
Task Type
Benchmark
spatial
temporal
spatial
temporal
DrawBench PartiPrompts
X
Texe2lmage
X
UCF-101
X X
X
X
Kinetics
X
X
XX
X
Video
MSR-VTI
/
X
X
X
X
Make-a- Make-a-Video-Evee Video-Eval
X
X
FETV (Ours)
aspects of video generation. (2) Lack of reliable automatic evaluation metries. In text-to-image generation, it has been found that the existing automatic evaluation metrics are inconsistent with
human judgment 66 27 28 However, in T2V generation, this problem is still under-explored.
To address these issues, we introduce a benchmark called FETV for the Fine-grained Evaluation of Text-to- Video generation, which offers a comprehensive categorization system for T2V evaluation. FETV includes diverse set of text prompts, categorized based on three orthogonal aspects: major content, attribute control, and prompt complexity (see Figure D. We also propose several temporal categories specifically designed for video generation, encompassing both spatial and temporal content. As shown in Table compared with existing benchmarks, FETV highlights the most comprehensive categorization for fine-grainet evaluation of T2V generation. To efficiently categorize the text prompts, we employ a two-step annotation process involving an automatic assignment of category labels followed by manual review. On this basis, collect 619 categorized text prompts from existing open-domain text-video datasets 48 and manually- wwritte unusual prompts.
Using FETV, we perform comprehensive manual evaluations for four representative T2V models, enabling fine-grained analysis of their strengths and weaknesses across different prompt categories. The results highlight the weaknesses of existing T2V models in terms of (1) poor video quality for action and kinetic motion related videos and (2) inability to control quantity, motion direction and event order (see Section 3.1 for detailed definition of these categories). Furthermore, FETV can serve as testbed for assessing the reliability of automatic evaluation metrics through correlation analysis with manual evaluations. Our experiments reveal that the widely used automatic metrics of video quality (FID [T] and FVD (40) and video-text alignment (CLIPScore (T0) correlate poorly with human evaluation. response to this problem, we explore several solutions to improve CLIPScore and FVD. Based on UMT (22), an advanced
model (VLM), develop FVD-UMT for video quality and UMTScore for rideo-text alignment, which exhibit much higher correlation with humans than existing metries.
The contributions of this work are as follows: (1) We propose the FETV for fine- fine-grained evaluation of open-domain T2V generation. Compared with existing benchmarks, FETV provides a more comprehensive view of T2V models' capabilities from multiple aspects. (2) Based on FETV, we conduct a comprehensive manual evaluation of four representative open T2V models and reveal their pros and cons from different aspects. (3) We extend FETV testbed to evaluate the reliability of automatic T2V metries and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics.
2 Related Work
Benchmarks for Text-to-Video Generation. Most existing studies on T2V generation adopt datasets from other related tasks to evaluate their models. UCF-101 (88 and 4inetics-400/600/700
[6 4. (36] are initially proposed for the task of human action recognition, which cannot cover the diversity of open-domain video generation. MSR-VTT [48), which is commonly used for video captioning and video-text retrieval, consists of text-video pairs covering 20 categories with diverse video contents. However, its categorization only considers the topic of videos (e.g., "music", "sports" and "news") while ignoring other important aspects in text-to-video generation, e.g., the temporal

---

Major
Major Content
people; sconirry & natural object
light change
Attribuze Control
Young enjoying
All Test
new of mouniain
Tansord
Prompts
fores in merning during
speed
suense. sow motion.
Prompt Complexxity
Prompt Complexity
complex
Simple
Camble
(4) Categorization of Single Data
(c) Categorization Test Set for Fine-Grained Evaluation
Spatia buiddigs infratracture: foce bewerage:
Spatal
Temporal
Prompt Complexity
lifh: changi:: fiid mations
actionn kineeic
Color quant
view
speed; midion dieccion: evant
single, mediati complex
(b) Summary of All Categories
Figure 1: Illustration of multi-aspect categorization (a-b), based which can realize fine- grained evaluation (c). Details of each category are shown in Figure and Appendix B.
contents and the attributes text prompt aims to control. In text-to-image generation, DrawBench 31 and PartiPrompts (50) introduce another aspect of categorization concerning the type of challenge prompt involves. However, their prompts lack descriptions of temporal content, which are not suitable for T2V generation. Make-a-Video-Eil [34] contains 300 text prompts specially composed for T2V generation. However, their categorization (i.e., animals, fantasy, people, nature and scenes, food and beverage) only considers spatial content.
Automatic Metries for Text-to- Video Generation. There are two types of automatic open T2V metrics, which evaluate the video quality and video-text alignment, respectively. For video quality, the widely used metrics are the Inception Score (IS) 32 Frechet Inception Distance (FID) [1. and Frechet Video Distance (FVD) 40). Based on a classification model, IS assesses whether every single generated video contains identifiable objects and whether all the videos are diverse in the generated objects. FID and FVD compute the distribution similarity between real and generated videos in the latent space deep neural network. In terms of video-text alignment, existing works on open T2V mainly adopt the CLIPScore [T0] CLIPScore is initially proposed to measure the similarity between machine-esn generated task of text-to-image generation, it has been found that the automatic image quality and image-text alignment metrics are inconsistent with human judgment 27 28 this problem remains under-explored.
machine-generated captions and the given image, using the pre- pre-trained CLIP model [29]. In the
When it comes to open T2V,
Models for Text-to- Video Generation. Existing text-to-video generation models can be divided into the Transformer-basee models 45
and the Diffusion-basser models B3 51 17 25 2 9 44 The former first converts the videos into discrete visual tokens and then trains the Transformer model generate these visual tokens. The latter generates videos using the Diffusion
46 13
Models [37 12 49]. either on the original pixel space or a latent space. In this paper, We evaluate three models that are open-sourced up to the time of writing this paper, namely Cog Video [T5]. Text2Video-zen [17] and ModelScopeTT T2V 44, as well as more recent model ZeroScope [39 Note that text-driven video editing 226 47 where the input a text prompt and source video, may also be referred to text-to- toi-tt--oee video generation. In this paper, we focus the typical kind of text-to-video generation, where the input condition is only a piece text prompt.
3 FETV Benchmark
FETV is dataset of text prompts divided into different categories from multiple aspects for fine- grained evaluation of T2V generation. In this section, we will introduce the multi multi-aspect categoriza- tion, how collect the prompts and category labels and the application scope of FETV.

---

Actions
Kinetic Motions
Prompts that invalve actions people or animals.
Description
Prompis that involve the motion of solid objects.
Example Prompts
people ure doncing: horse eating
Example
the cara drove fast:
Prompts
DNA strands
Fluid Motions
Light Change
Description
Prompts
like fluids, egg, the deformation of objects.
Description
Prompts
involve change of light or color over time.
may
Example Prompts
Time maving clouds with
Example Prompts
frre burning; lopso, mowig pcross the night sky
the water, falls down to the cliff really fast:
Temporal categories under the "major content" aspect.
Speed
Metion Direction
Event Order
Desctarics
Desotorce
Dasotation
Prompts that invee cantist over order everts
the cas drow/oos Sing abermatio
Exarple
Swap couete oor cround loree youkle
Fermse
stolo hand ThaT and Honn they fig voch obler
Promusts
(b) Temporal categories under the "attribute control" aspect.
Figure 2: Descriptions and examples of the temporal categories.
3.1 Multi-Aspect Categorization
As shown in Figure I each prompt is categorized based on three aspects, namely the major content describes, the attribute aims to control in the generation videos, and complexity. The "major content" aspect is further divided into spatial and temporal categories, the former focus the type of objects described in the prompt, while the latter focus on the type of temporal content. Similarly, the "attribute control" aspect is comprised of both spatial attributes and temporal attributes. The "prompt complexity" aspect has three levels of complexity, namely "simple", "medium", and "complex", depending the number of non-stop words contained in prompt. With the multi-aspect categorization every single prompt, the entire FETV benchmark can be divided into different subsets (Figure I (c)), therefore enabling fine- grained evaluation. Figure prreeents the descriptions and corresponding examples of the temporal categories. The complete description of all categories can be found in Appendix B.T
3.2 Data Collection
Prompt Sources. The prompts of FETV come from two kinds of sources. (1) we reuse the texts from existing datasets of open-domain text-video pairs, which guarantees the diversity of prompts. Specifically, we use the MSR-VTT 48 test set and WebVid [ validation set, which contain 59,800 and 4,999 text-video pairs, respectively. (2) we manually write prompts describing scenarios that unusual in the real world, which cannot be found in existing text-video datasets. These "unusual prompts" are inspired by the "Imagination" category in PartiPrompts (50) and "Conflicting" category in DrawBench (3I, which aims to test the generalization ability of the generation models. To write unusual prompts, we first define several types of unusual cases, namely "object-action", "object- object", "object-color", "object-quantin"" example, the prompt "A cat is driving a car." belongs to the case of unusual "object-action" actionn.. Then, for each case, enumerate several specific objects and attributes and write the prompts.
"object-direction" "object-speed" and "event order". For
Prompt Categorization and Selection For the prompts from both sources, we assign category labels to them based on our multi-aspect categorization structure. This is achieved in two steps: In the first step, we define list of keyphrases, WordNet synsets and some hand-crafted matching patterns for each category and automatically categorize the prompts. For example, to determine

---

Spatial Major Content
"emporal Major
480
Sample Count
Sample Count
307
232
282
16
Peocle
arimads
retrile
plants
grttle
muiding
pcenery
Blostrations
adtond
Wiitts motions
mations
light change
Prompt Complexity
330
Spatial Attribute Control
Temporal Attribute Control
Sample Countt
250-
Sample Count
Sample Count
116
383
66
34
color
Quaitity
camara view
sp22d
modion direction evant orber
Figure 4: Data distribution over categories under the "prompt com- plexity" aspect.
Figure 3: Data distribution over categories under the "major content" (upper) and "attribute control" (lower) aspects.
whether a prompt belongs the "animals" category, we iterate over the words in the prompt and see whether any of them belong to the animal-related WordNet synset. Details of the automatic matching rules for each category are presented Appendix G In the second step, we manually select prompts for each category and make some revisions to the category labels to ensure that they are correct. Specifically, we enumerate combinations of (ca, cy), where G₁ belongs the six "attribute control" categories and Cj belongs to the four "temporal major content" categories. For each combination, we select roughly 20 prompts that belong to both e and cj. according to the first-step categorization, and then revise the incorrect category labels. For the text prompts from existing datasets, we also revise the ones that do not align well with the reference video.
3.3 Dataset Statistics and Data Format
We collect total of 619 prompts with corresponding categorizations, among which 541 prompts are from existing datasets and 78 unusual prompts are written by us. Such data scale comparable with existing text-to-image/vider benchmarks, e.g., DrawBench (200 prompts), PartiPrompts (1,600 prompts) and Make-a- Make-a-Video-Ein Video-Eval (300 prompts), which are mainly proposed for manual evaluation. The data distributions over categories under different aspects illustrated in Figure and Figure 4 which show that FETV contains enough data to cover the diverse range of categories. We also compare FETV with MSR-VTT and Web in terms of the data distribution over categories, which can be found in Appendix B.2
Each data sample in FETV consists of three elements: the prompt t, the reference video ef (not available for the unusual prompts) and the categorization labels {C"} .A is the collection of the three aspects, C" is the categorization labels of under aspect a. Taking the prompt in Figure T example, when the aspect is "attribute control", C = ("speed").. Note that except for the complexity aspect, we allow each prompt to be classified into multiple categories under the same aspect. This setting more realistic compared with existing benchmarks that only classify each prompt into a single category. More examples of FETV data can be found Appendix B.3
3.4 Application Scope
Theoretically, FETV can be used for both manual and automatic evaluation. However, considering that existing automatic T2V metrics correlate poorly with humans (see Section 5). we mainly rely on manual evaluation to obtain more accurate results. In addition evaluating T2V models, FETV can also be extended to diagnose automatic metrics, using the manual evaluation as reference.
5

---

4
Manual Evaluation of Open T2V Models
4.1 Models
We select four representative T2V models, which are open-sourced, for evaluation. We briefly describe them as follows and defer the details to Appendix [
CogVideo [3 is based on the Transformer architecture 42]. It first transforms the video frames into
discrete visual tokens using VQVAE [41] and then utilizes two Transformer models to hierarchically generate the visual tokens from a given text prompt.
Vixeozideooeoo 17 is based on the Diffusion Probabilistic Model 37 30 with motion dynamies,
It generates videos by
enhancing the latent codes of a text-to-image Stable Diffusion model without training the model on any video data.
ModelScope T2V [44] another diffusion- model. It extends the T2I Stable Diffusion model with temporal convolution and attention blocks and conducts training using both image-text and video-text datasets.
ZeroScope (39] is watermark-free T2V generation model trained from the weights of Mod- eIScopeT2V. It is optimized for 16:9 compositions.
4.2 Evaluation Setups
Evaluation Criteria. We evaluate the generated videos mainly from four perspectives: Static quality focuses on the visual quality of single video frames. Temporal quality focuses the temporal coherence of video frames. Overall alignment measures the overall alignment between video and the given regarding specific attributes. For the first three perspectives, we score the videos hased on a 1-5 Likert- scale judgement. For fine-grate fine-grained alignment, we adopt 1-3 Likert-scale judgement. To facilitate the consistency between human evaluators, we carefully design the wording of questions and descriptions of each rating level. The agreement between our human evaluators is high, with Krippendorff's
prompt. Fine-grained alignmen measures the video-text alignment
[19] of 0.711, 0.770, 0.638 and 0.653 in the above four perspectives. In Appendix E] we summarize the details of our instruction document for manual evaluation and the annotation interface.
Implementation Details. We generate three videos for each prompt using the three models respec- tively. For each generated video and each of the above four perspectives, we collect three rating scores from graduate students in our laboratory and report the average rating. For the generated videos, we obtain 619 samples x 3 models x perspectives x 3 humans =16,713 1,3 ratings for the first three perspectives and 5,184 ratings of fine-graines alignment perspective (note that not every prompt involves an attribute for evaluating fine-graine alignment). We also evaluate the 541 reference videos, which produce 6,219 ratings for the four perspectives. In total, we collect 28,1 16 ratings in our manual evaluation. To obtain the performance under specific category, we report the average rating of all videos whose prompts belong to this category.
4.3 Results and Analysis
In this section, we present and analyse the results from a quantitative perspective (some qualitative analyses are shown in (p.5)ddiF..5).... Specifically, are interested in the following questions: (I) how video quality varies when generating different major contents, (2) how video-text alignment varies when controlling different attributes facing different prompt complexity. We also investigate (3) how well the models generalize to unusual prompts in Appendix F4
The results of video quality are presented in Figure We can observe that: (1) In terms of the spatial categories, the generated videos containing "people" and "animals" have poor quality, while the videos containing "plants" and "scenery & natural objects" are of good quality. This is because videos of the former two categories involve many high-frequency details (e.g., human faces and fingers) which are more difficult to learn than the low-frequency information in the latter two categories (e.g., water and clouds). (2) When it comes to the temporal categories, the videos about "actions" and "kinetic motions" are of worse quality, since the corresponding temporal content is of higher frequency. Although there is a correlation between some spatial and temporal categories
We use the dreamlike-photoreall 2.0 version, following the default setup of Vexe2
'In this context, we reuse the word tine-graines to describe alignment between single video-text pair, while in other contexts it refers to the evaluation of T2V models on specific categories of prompts.

---

static quality
temporal quality Animals
Animals
People
Scenery
People
Scenery
Plants
Food Plants
Pood
Illustrations
Building
Illustrations
Building
Artifacts
Vehicles
Artifacts
Vehicles
ZeroScape
(a) Results across spatial major contents.
static quality Fluid Motions
temporal quality
Fluid Motions
Light Change
Kinetic
LigHt Change
Kihetic Motions
Mations
Action
Ground-Truth
ModeSSape T2V
CogVideo
Text2Videezeree
ZeroScape
(b) Results across temporal major contents.
Figure 5: Manual evaluation of static and temporal video quality.
overall alignment
fine-grained alig amint
Color
Color
overall alignment
4.5
simplc
Quantity
4.0
Event Order
Quantity
Event Order
4.5
4.5
Camieaa
Motion Direction
Carrerra View
Motion
View
Delection
medium
complex
Speed
Speed
Geound-huut
ModelSeopeT2V
CogViden
Zendsoop
(a) Results across attributes to control.
(b) Results across complexity.
Figure 6: Manual evaluation of video-text alignment.
(e.g., "action"" is related to "people"), our temporal categorization provides more direct view of the model performance from the temporal content perspective. (3) None of the four models achieve comparable video quality with real videos, especially in the aforementioned poor-quality categories. (4) Text2Video-zer performs the best in static quality while lagging behind in temporal quality. This is because Text2Video-7er inherits the image generation power of Stable Diffusion, but the absence of video training prevents it from generating temporally coherent content. (5) ZeroScope, which is trained from ModelScopeT2V, is comparable with ModelScopeT2V in temporal quality
7

---

Rank
Model
Score
Rank
Model
Score
Rank
Model
Score
TextVVieeo-erro
4.11
ModelScopeT2V
4.07
ModelScopeT2v
3.79
ModelScopeT2V
3.78
ZeroScope
3.72
ZeroScope
3.57
ZeroScope
3.52
CogVideo
3.30
Text2Video-Zer
3.42
CogVideo
TextZVideo-Zero
CogVideo
3.10
(a) Static Quality
(b) Temporal Quality
(c) Overall Alignment
Figure 7: Leaderboard on FETV benchmark based on manual evaluation.
while performing worse static quality. We conjecture that this because the ZeroScope is primarily optimized for the 16:9 composition and watermark-free videos, instead of for better visual quality and video-text alignment.
The results of video-text alignment are shown in Figure 6 We can see that: (I) Existing T2V models can already control "color" and "camera view" in most cases. (2) All four model struggle to accurately control "quantity", 'motion direction"" and "event order". However, ModelScope exhibits some preliminary ability to control "speed" and "motion direction". which is not observed in CogVider and Tixe-Video-zero. (3) ZeroScope slightly underperforms the original ModelScope in video-text alignment. We attribute this to the same underlying cause as explained for the video quality performance. (4) The relative performance among different attributes and T2V models similar between fine-grained and overall alignment, with a few exceptions. This is because the overall alignment may inevitably be affected by other attributes. Therefore, the fine-grained alignment is more accurate when evaluating controllability over specific attributes, but the overall alignment an acceptable approximation. (5) While video-text alignment is slightly lower for the "complex" prompts, cannot observe clear correlation between prompt complexity and video-text alignment.
In addition to the fine-grained evaluating results, we also summarize the leaderboard on the entire FETV benchmark in Figure
5 Diagnosis of Automatic Evaluation Metrics
In this section, we perform automatic evaluations on FETV and diagnose their correlation with manual evaluations.
5.1 Metrics
For video-text alignment, we investigate five metrics. (1) CLIPScore [0 is widely used to evaluate T2V/I generation models. CLIPScore based on CLIP 29 vision-language model (VLM) pre- trained on large-scale text-image pairs. (2) CLIPScore-FE ft is based on a CLIP model fine-tuned on the video-text retrieval task using MSR VTT data. (3) BLIPScore and (4) UMTScore replace CLIP with more advanced VLMs, i.e., BLIP 21 and UMT 22). BLIP adopts bootstrapping language-iman pre-training. UMT pre-trained large-scale video-text data and we adopt the fine-tuned version on MSR-VTT. Then, we compute BL.IPScore in the same way as CLIPScore. For UMT, we utilize the cross-modal decoder's video-text matching result and refer to this metric as UMTScore. We also adopt a Video Large Language Models (L.LMs)), i.e., Otter 20), to evaluate video-text alignment by formulating it as a Video QA task. Specifically, we first extract the key elements from the text prompt and generate corresponding yes-no questions via the Vicuna model [5). Then, we feed the generated (or ground-truth) videos and questions into the Video LLM. The average number of questions that receive a positive answer is defined as the alignment score for a specific video-text pair. We name the Video L.M--asee metric as (5) Ott.
For video quality metries, we investigate FID 1I and FVD 40 I3D model 3 as video encoder. We introduce FVD-UMT, an enhanced version of FVD that leverages the UMT vision encoder for video feature extraction. More details of these metrics and our implementations can be found in Appendix
The original FVD adopts

---

Table 2: Correlation between automatic and manual evaluations of video-text alignment, measured by Kendall's Ta (left) and Spearman's 's (right) coefficients. We also report the average inter-human correlation in the final row as reference.
Camera View
Motion
Event Order
Color
Quantity
Speed
All
Direction
CLIPScore
0.157/0.218
0.147/0.202
0.254/0.344 345
0.178/0.24
0.141/0. 194
0.177/0.248
0.177/0.243
CLIPScore-tt
0,206/0.287
0.250/0.340
0.293/0.402
0220/0.280
0.254/0. 348
0.157/0.221
0,224/0.309
BLIPScore
0.222/0.307
0,207/0.282
0.285/0.394
0.195/0.260
0.223/0. 305
0.180/0.22
0.235/0.322
Otter- VQA
0.049/0.070
0.134/0.188
0.027/0.038
0.051/0.073
0.119/0.11 166
0.146/0.206
0,081/0.114
UMTSScore
0.304/0.420
0.394/0.528
0.300/0.415
0.296/0.407
0.356/0.476
0.295/0.406
0.309/0.425
Human
0.547/0.702
0.647/0.784
0.447/0.595
0.539/0.687
0.619/0.747
0.517/0.680
0.576/0.719
cre t
0.31
CLIPScore-It
0.30
9.29
BLIPScore
Otter-WAA T
0.70
25
0.30
0.28-
0.44
0.29
0.62
0.26-
1.0
4.5
4.0
Human
Human
Human
Human
CopVides
Mods
Teci2Vibe--s
conoccpp
Ground Truch
Figure 8: Automatii and human ranking of the T2V models in terms of video-text alignment. Results on specific categories are reported in Appendix F.1
5.2 Evaluation Setups
Video-Text Alignment Metries. We use the same generated and reference videos for manual evaluation (one video for text prompt using one model) and compute the video-text alignment for each video-text pair. Then, we measure the correlation between automatic and manual evaluations, using Kendall's тe coefficient and Spearman's 'ss p coefficient. To obtain the correlation under specific category, we collect the manual and automatic evaluation results of video-text pairs that belong to this category and compute the coefficients.
Video Quality Metrics. Different from the video-text alignment metrics, FID and FVD evaluate the quality of group of generated videos instead of a single video. To evaluate the fine-grained video quality in specific categories, for each open T2V model, generate 300 videos for each category under the "major content" aspect, based on FETV prompts. In this case, each prompt can be used multiple times but the generated videos are different due to the randomness the diffusion process and visual token sampling. We also sample 300 reference videos for each category. This is achieved by using the automatic categorization results of MSR-VT VTT and WebVid prompts (introduced in Section (.2) and collecting the corresponding videos. To compute the overall FID and FVD for T2V model, we sample 1,024 generated and reference videos (the impact of video sample number is analyzed in Appendix F.3
The results are averaged across 4 runs with different random seeds.
5.3 Results and Analysis
Video-Text Alignment Metrics. Table 2 shows the correlation between automatic and manual evaluations. We can see that: (1) Although the widely-used CLIPScore is positively correlated with manual evaluations, the overall correlation degree is very weak compared with inter-human correlations. (2) Fine-tuning CLIP on video-text retrieval and replacing CL.IP with BLIP are beneficial, which promote the correlation across almost all categories. However, CLIPScore-ft and BLIPScore are still inconsistent with the human ranking of videos generated from different models, as shown in Figure Therefore, we cannot rely on CLIPScore and BLIPScore to accurately evaluate open T2V models. (3) UMTScore exhibits the strongest correlation with humans when measured by Kendall's To and Spearman's p. More importantly. consistently aligns with human judgments in the T2V model rankings. The performance of UMTScore suggests that training VLMs with video data is important and the evaluation of open T2V models can benefit from the development of stronger VLMs. (4) The correlation varies across different "attribute control* categories. It is interesting to see that for UMTScore, the correlation in the challenging categories (for open T2V models), i.e., "Quantity" and "Motions Direction". is higher than in the relatively simple "Color" and "Camera

---

Prompt
men leading a donkey across a field towards a parked truck
Generate
Ground Truth
Human
2r0)
2.0 3ro)
3. (0222))
4.57
UMTSscree
0.13813rdt
2.527)
BLIPScore
0,407/3rd)
0,367/4t)
0,42/1stt
0.478(2nd)
CLISScore-tt
0:288(41))
0.334)3rd)
6.333/1
0.314/2nd)
CUPScore
(3rd
2nd06(2nd)
0.337
0.284 4th0
Otter-VOA
0.767(18))
0. (.6
0.633/400)
0.867(2nd)
Figure 9: Video examples and alignment scores measured by different metrics, The left three videos are generated and the rightmost video is the ground-truth More examples are shown in Appendix F.6
900
80
EVD-JMT
700
200
600
70
190-
530
420
180
3.2
CogMidee
ModeeScopeT2V
Text2VV ded-uper
Saro
Figure 10: Automatic and human ranking of T2V models in terms of video quality (averaged static and temporal quality). Results on specific categories are reported in Appendix F.2
View". We conjecture that this is because, in the simpler categories, the performance of open T2V models more similar and it is more difficult distinguish the difference, and vice versa.
Figure 9 presents a case study of video-text alignment scores. We can see that the results measured by CLIPScore and Otter-VO VQA strongly differ from human evaluation. CL.IPScore-f ft and BI RISPee correlate better with humans, while still mistakenly assigning lower score to the ground-truth video. In comparison, UMTScorees ranking of video-text alignment is most consistent with humans.
Video Quality Metrics. For FID and FVD, we cannot calculate the correlation coefficients because they are not sample-wise metrics. investigate their correlation with humans, we focus on system-wise ranking of open T2V models. As we can see in Figure TO, the FID and the original FVD with I3D video encoder are loosely coupled with the human's perception of the video quality. This result reveals the critical problem of using FID and FVD as measurement of video quality, which is common previous works. In comparison, the FVD-UMT metric, which is enhanced with the UMT model video feature extractor, generates consistent rankings with humans.
Conclusions
In this paper, we propose the FETV benchmark for fine-grained evaluation of open-domain text-to- video generation. Compared with existing benchmarks, FETV is both multi-aspect and temporal- aware, which can provide more specific information the performance of T2V models. Based on FETV, we conduct a comprehensive manual evaluation of representative T2V models and reveal their pros and cons in different categories of prompts from different aspects. Furthermore, we extend FETV to diagnose the reliability of automatic T2V metrics, revealing the poor correlation of existing metrics with humans. To address this problem, develop automatic metrics using an advanced VLM called UMT, which can produce a consistent ranking of T2V models with humans.
7 Limitations and Future Work
The limitations of this work can be viewed from two perspectives. First, while the proposed UMT- based metrics are more reliable than existing automatic metrics, they still have room for improvement to better align with humans. Second, due to the lack of open-sourced open T2V models, the number of models evaluated this work limited. We invite follow-up studies to evaluate their models on benchmark and encourage projects to open-source their models.
10

---

Acknowledgments and Disclosure of Funding
We thank all the anonymous reviewers for their constructive comments. This work is supported in part by Huawei Research Grant and National Natural Science Foundation of China (No. 62176002). Xu Sun is the corresponding author of this paper.
References
[1] M. Bain, Nagrani, G. Varol, and Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In ICCV, pages 1708-1718, IEEE, 2021.
[2] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[3] J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CVPR, pages 4724-4733. IEEE Computer Society, 2017.
[4] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. CoRR, abs/1808 808.01340, 2018.
[5] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng. Z. Wu, H. Zhang, L. Zheng. S. Zhuang, Y. Zhuang. J. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023, URL htttts://mmss:: hwtps//o/o//bo//////////// vinnaa/
[6] M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In NeurlPS, 2022.
[7] A. Dosovitskiy, L.. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16
words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[8] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV (4), volume 12349 of Lecture Notes in Computer Science, pages 214-229. Springer, 2020.
[9] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen. Latent video diffusion models for high-fidelity long video generation. CoRR, abs/221 1. 1.13221, 2022.
[10] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free
evaluation metric for image captioning. In EMNLP (1), pages 7514-7528. Association for Computational Linguistics, 2021.
[11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, pages 6626-6637, 2017.
[12] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[13] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models. CoRR, abs/2210.02300 2022.
[14] J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. In NeurlPS, 2022.
[15] W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. CORR, abs/2205. 15868, 2022.
[16] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset. CoRR, abs/1 abs/1105500650, 2017.
[17] L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. CoRR, abs/2303. 2023.

---

[18] J. Kim, Y. Kim, J. Lee, K. M. Yoo, and S. Lee. Mutual information divergence: A unified metric for multimodal generative models. In NeurlPS, 2022.
[19] K. Klaus. Content analysis: An introduction to its methodology. 1980.
[20] B. Li, Y. Zhang, Chen, J. Wang. Yang, and Z. Liu. Otter: A multi-moda model with in-context instruction tuning. CoRR, abs/2305.0372 2023.
[21] J. D. Li, C. Xiong, and S. C. H. Hoi. BLIP: bootstrapping eoeguage-image pre- pre-training for unified vision-language understanding and generation. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 18888-12900 PMLR, 2022.
[22] K. Li, Y. Wang, Y. Li, Y. Wang, Y. He, L. Wang, and Y. Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, 2023.
[23] T. Lin, M. Maire, S. J. Belongie, J. Hays, Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In ECCV volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014.
[24] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293-304, 2022.
[25] Z. Luo, D. Chen, Y. Zhang, Y. Huang. L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan. Videofu- sion: Decomposed diffusion models for high- video generation. CoRR, abs/2303.083200 2023.
[26] E. Molad, E. Horwitz, D. Valevski, A. R. Acha, Y. Matias, Y. Pritch, Y. Leviathan, and Y. Hoshen. Dreamix: Video diffusion models are general video editors. CoRR, abs/2302.01329, 2023.
[27] M. Otani, R. Togashi, Y. Sawai, R. Ishigami, Y. Nakashima, Rahtu, J. Heikkilä, and S. Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. In CVPR. IEEE, 2023.
[28] G. Parmar, R. Zhang, and Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In CVPR, pages 1 11400-1141 11410. IEFE, 2022.
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Muchine Learning Research, pages 8748-8763. PMLR, 2021.
[30] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High- nigh-resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685. IEEE, 2022.
[31] C. Saharia, W. Chan, S. Saxena, L. J. Whang, E. L. Denton, S. K. S. Ghasemipour, R. G. Lopes, B. K. Ayan, Salimans, Ho, J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[32] T. Salimans, J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. NIPS, pages 2226-2234, 2016.
[33] C. Schuhmann, R. Beaumont, Vencu, Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
[34] U. Singer, A. Polyak, T. Hayes, X. Yin, An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, D. Parikh, S. Gupta, and Y. Taigman. Make-a-video: Text-to-video generation without text-video data. CoRR, 01022009.1479... 2022.
[35] I. the price, image quality and perks of stylegan2. In CVPR, pages 3616-3626. IEEE, 2022.
Skorokhodov, S. Tulyakov, and M. Elhoseiny. Stylegan-v: A continuous video generator with
[36] I.. Smaira, J. Carreira, E. Noland, F. Clancy, A. Wu, and A. Zisserman. A short note on the Sinetics-700-222 human action dataset. CoRR, abs/2010. 10864, 2020.
12

---

[37] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In JCML, volume 37 of JMLR Workshop and Conference Proceedings, pages 2256-2265. JMI.R.rrg 2015.
[38] K. Soomro, A. R. Zamir, and M. Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402 2012.
[39] S. Sterling. zeroscope-v2, June 2023.
[40] Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, FVD: A new metric for video generation. In DGS@ICLR, OpenReview.net, 2019.
[41] A. van den Oord, O. Vinyals, and Kavukcuoglu. Neural discrete representation learning. In NIPS, pages 3306-6315, 2017.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention all you need. NIPS, pages 5998-6008, 2017.
[43] R. Villegas, M. Babaeizadeh, P. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual description. CoRR, abs/2210.02399, 2022.
[44] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang. Modelscope text-to-video technical report. CoRR, abs/2308 065766651111 2023.
[45] C. Wu. L. Huang, Q. Zhang, B. Li, Ji, F. Yang. Sapiro, and Duan. GODIVA: generating open-domain videos from natural descriptions. CoRR, abs/2104. 14806, 2021.
[46] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV (16), volume 13676 of Lecture Notes in Computer Science, pages 720-736 Springer, 2022.
[47] J.Z. Wu, Y. Ge, X. Wang, W. Lei, Y. Gu, W. Hsu, Shan, X. Qie, and M. Z. Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. CoRR, abs/2212.11565 2022.
[48] Xu, T. Mei, Yao, and Rui. MSR-VTT: large video description dataset for bridging video and language. In CVPR, pages 5288-5296. IEEE Computer Society, 2016.
[49] R. Yang, P. Srivastava, and S. Mandt. Diffusion probabilistic modeling for video generation. CORR, 05s/2203.094811 2022.
[50] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang. J. Baldridge, and Y. Wu. Scaling autoregressive models for content-rich text-to-image generation. Trans. Mach. Learn. Res., 2022, 2022.
[51] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng. Magicvideo: Efficient video generation with latent diffusion models. CoRR, abs/2211. 121221111111 2022.
13

---

A
Overview of Appendix
More Details of FETV.
Text-to-Video Generation Model Details.
D
More Details of Automatic Evaluation Setups.
More Details of Manual Evaluation Setups.
More Experimental Results.
Automatic Categorization Rules.
Licensing, Hosting and Maintenance Plan.
Datasheet.
B More Details of FETV
B.1 Category Descriptions
In Figure the main paper, illustrate the descriptions and examples of the temporal categories. In this appendix, Figure TI presents the descriptions and examples of the spatial categories under the "major content" and "attribute control" aspects. Table presents the descriptions and examples of different prompt complexity levels.
B.2 Data Distribution
Figure 12 compares FETV with MSR-VTT and WebVid in terms of their data distribution categories. To obtain the distribution of MSR-VTT and WebVid, we randomly sample 100 data from each dataset and manually annotate the categories exhibits more uniform distribution over the categories, while MSR-VTT and WebVid are more biased toward certain categories. Particularly, more than 60% of the MSR-VTT and WebVid prompts do not involve the six attributes introduced in FETV. Consequently, the ability to control these categories, especially the challenging ones, cannot be reflected by the performance on MSR-VTI VTT and Web Vid (note that these two datasets lack categorization of the attributes).
construct FETV. As we can see, FETV
B.3 Data Examples
Table shows FETV prompt examples with corresponding categorizations, which cover all 22 categories under the three aspects.
Text-to- Video Generation Model Details
C.1 Video
Model Architecture. Video 15 generates videos as discrete visual tokens, encoded by the VQVAE[4]) using the Transformer model 42
The generation process consists of two stages. In the first stage, a Transformer sequentially generates some keyframes based on the text prompts. In the second stage, another Transformer generates the remaining frames by interpolating the generated frames recursive way. both stages, Cog Video can generate videos according to a specified frame rate. Cog Video is initialized from the pre-trained text-to-image model Cog View2 (6] and inserts a temporal channel attention to model the dynamics in videos. In total, the Transformer of both stages has 9.4 billion parameters, among which billion parameters are from Cog View2.
Training Setups. Cog is trained a private dataset of 5.4 million captioned videos with 160x 160 spatial resolution. The first-stage Transformer is trained with 1 fps frame rate and the second-stage Transformer is trained with 2,4,8 fps frame rates.
For inference of CogVideo, we adopt the official implementation with setups. We use the Transformer models of both stages and generate videos with 33 frames (a duration of seconds), with spatial resolution of x80x (upsampled by Cog View2). The inference of Video consumes approximately 24 GB of GPU memory. We use two types of GPU, i.e., Nvidia A40 and TTTAN RTX. The inference of all four T2V models run on single GPU. The codes are based on the Pytorch framework
Implementation Details.
default
14

---

People
Animals
Descriptonn
Prompts that involve people.
Description
Prompts that involve animals.
Example Prompts
people are doncing: gli pirgting
roé6its are running sroundy horse is eoting
Prompts
Artifacts
Description
Description
Prompts that invoive human objects, including large
Example Prompts
Coast forest view trrough water surt Time lapse of white flyacinth flowers bleoming.
Example Prompts
A mar typing on wirniess koyboord. close up, The musicion plays the guitar rinse
Vehicles
Buildings & Infrastructures
Prompts that involve vehicles.
Description
Prompts that Invalve buildings and infrastructures.
Example Prompot
the cars drowe fast; lugbost curgo ship
Example Prompts
City skyine and space seedl sunrise time lopse; sydney bridge
Food & Beverage
Scenery & Natural Object
Description
Prompts invalve food beverage.
Descrixtion
Prompts that invoive lifeless natural objects and
Tooseing with beet, close up weew; womon chepse a green pepper and on onion
Example Prompts
citf really fast, Time lyuse moving ciouds with bize sky
Prompos
Illustration
Promots that involve graphical representations, including geometrical objects and symbols.
Oscilating lamiitous particles form suyfoce smooth arimation looped, randion giowing purple and pint nnon liees
Example Prompts
(a) Spatial categories under the "major content" aspect.
Color
Camera View
Description
Prompts that invaive control over color:
Description
Prompts that involve control over camera view.
Example Promots
someone driving un orange sports car: womon chops gresn oepper and on onion
Bample Prompts
fist person view of man drivitg red formula one car: overlyod view pingpong players compete the tahll
Quantity
Prompts that quantity.
Exemple Prompts
two young girk playing with or horse: three eating
(b) Spatial categories under the "attribute control" aspect.
Figure 11: Descriptions and examples of the spatial categories.
15

---

Spatial Content
Temporal Content
Attributes
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0.0
Reople
Arttects
Soneeyy
Buading
Plente
Animald
Food
Buarrdao
Action
Kistte Musens
Minones
Home
Ohange
Color
Camera
View
Mosion cieesim
Spred
Quantyy
Ordker
(a) Data distrihution of 200 random samples from MSR-VTT and Web Vid.
Spatial Content
Temporal Content
Attributes
0.4
0.20
0.15
0.3
0.2
0.1
0.1
0.0
Aeople
Artlacs
Sortwy
Buiting
Parvay
Asimals
Hove
mustratom
0.0
Anten
Muthin
Mechems
Hove
Light Crolage
0.00
Name
Colar
Comera
Viw
Directio
Stated
Quanity
Greeer
(b) Data distribution of FETV.
Figure 12: Percentile distribution of data across categories. "None" denotes the data samples that do not belong any the categories under certain aspects. Note that the distributions do not sum to one because each data sample can belong to multiple categories.
Table 3: Descriptions and examples of different "prompt complexity" categories.
Category
Description
Example Prompts
Promptl: the cars drove fust
Simple
Prompts that involve non-stop words.
Prompt:: people dancing
Prompt3: a dog is sneezing
Promptl: A mun is typing on wireless keybourd. cluse up. Prompe2: flying counter clockwise around large yacht
Medium
Prompts that involve ~ 8 non-stop words.
Prompt:: aerial surrise shot lown, flying forwards
Prompti: down slow motion wide tracking shot of man walking along story ground towardss clouds
Prompt2: in a cookery program the preparation of vegetable soup is shown cutling leaver
Complex
Prompts that involve more than 8 non-stop words.
add pinch of sult water and boiling Prompt:; a corn field moves quickly past the camera there is a shon city and then of double- passenger train
C.2 Text2Video-Zero
Model Architecture. Text2Video-Zero 17 adapts text-to-image generation models for zero- shot text-to-vided generation without any training video data. Specifically, it introduces three techniques: (1) injecting motion dynamics into the first frame's s latent code to obtain a sequence of dynamic frames; (2) replacing each frame's self-attention with cross-attention the first frame to enhance temporal coherence; (3) a background smoothing strategy to enhance the consistency of background across frames.
Implementation Details. Texr2Video-Zern is compatible with wide range of diffusion-basz text-to-image models. In this work, we utilize the drreamiike-photrenn version Stable Diffusion, following the default setup of Text2Video-Zero official implementation We generate videos of 2
16

---

Table 4: Prompts and categorizations of FETV data examples. "Fluid" denotes fluid motions; "Kinetic" denotes kinetic motions; "Light" denotes light change; "Buildings" denotes buildings & infrastructures; "Food" denotes food & beverage; "Scenery" denotes scenery & natural objects.
Major Content
Attribute Control
Prompt Complexity
Prompt
Spatial
Temporal
Spatial
Temporal
a person is cooking
People
Actions
None
None
Simple
rabbits rumnnn around someone folds paper mountain stream a fire burning sea side road driving orange sports girl pink dress fashion model walking ramp boys are eating barger four airplanes flying
Animals
Actions
None
None
Simple
Artifacts
Actions
None
Nome
Simple
Scenery
Fluid
None
None
Simple
Scenery
light
None
None
Simple
Scenery-Buillding
Fluid
None
None
Simple
People; Vehicles
Kineeic
Color
None
Medium
People:Artifacts
Actions:Kinetic
Color
None
Medium
Rows of young green sunflower plants in field in sunset, agriculture in spring
Plants
Light
Color
None
Complex
People:Food
Actions
Quantity
None
Simple
People;Vehicles
Actions:Kinetic
Quantity
Nome
Simple
a first person view of man driving red formala one car
People:Vehicles
Actions:Kinetic
Color;Canera
None
Complex
overhead view as pingpong players compete the table
People:Artifacts
Actions
Camera
None
Medium
Tousting with beer, close up view.
Food
Fluid
Camera
None
Simple
Bees flying, slow motion shot the cars drove fast Time lapse moving clouds with blue sky
Animals
Actions:Kinetic
None
Speed
Medium
Vehicles
Kinetic
None
Speed
Simple
Scenery
Fluid
Color
Speed
Medium
Flying counter clockwise around a large yacht
Vehicles
Kinetic
None
Direction
Medium
A wave approaching and crashing big rock shore.
Scenery
Fluid
None
Direction
Medium
man walks away from on suv
People:Vehicles
Actions:Kinetic
None
Direction
Simple
Abstract circular light pattern, artistic blurred background, comera mevvin from left right
Illustaaioons
Kinetic;Iiigt
None
Direction
Complex
an old man shakes hands with aoother and then they each other
People
Actions
None
Order
Medium
gayy places pan tin then places the into water bath
People:Aatifacts
Actions:Fluid
None
Order
Complex
drives mountain road and then on highway along the lake
Veheeles:Scenery: Buildings
Kinetic
None
Order
Medium
seconds fps, with spatial resolution of 512x512. The inference of Texe2 Zeroo consumes approximately 24 GB of GPU memory.
C.3 ModeIScopeTT2
Model Architecture. ModelScopeT22 44 extends the T2I Stable Diffusion model with temporal convolution and attention blocks, which enables video modelling. Then, the model is trained using both video-text and image-text datasets, which mitigates catastrophic forgetting of the T2I generation ability.
17

---

ModelScopeT22 is trained on both video-text pairs (Web Wid-V1d I and image-text pairs (LAION2B-en (3)). During training. 1/8 of GPUs are allocated to images and the remaining 7/8 GPUs are used for videos.
Training Setups.
Implementation Details. We implement ModelScopeT2y using the code and model checkpoints released in ModelScope) We generate videos of seconds at 8fps, with a spatial resolution of 256x256. The inference of ModelScopeT22 consumes approximately 16 GB of GPU memory.
C.4 ZeroScope
Model Architecture and Training Setups. ZeroScope [39 is watermark-free video generation model optimized for producing 16:9 compositions. It trained from the weights of ModelScopeT2V using 9,923 clips and 79,769 tagged frames at 24 frames, 576x320 resolution.
Implementation Details. We adopt the ZeroScope- mentation in HuggingFace. The generated videos are 3 seconds at 8fps, with a spatial resolution of 576x320. The inference of ZeroScope-v2 consumes upproximately 20 GB of GPU memory.
model following the official imple-
D More Details of Automatic Evaluation Setups
D.1 Video-Text Alignment Metrics
For the basic version of CLIPScore, adopt the implementation from [8]. with ViT-B/32 7 as the visual encoder backbone. For BLIPScore, we utilize the BLIP mode 21. with ViT-B/16 as the backbone, fine-tuned on the COCO dataset [23] for image-ter text retrieval. For CLIPScore-fi
fine-tune the ViT- 11-B//22 CLIP model for video-text retrieval on the 7k training split of MSR- VTT [8]
which has overlap with the reference videos in FETV. The implementation and
settings follow CLIP4CLIP [24]. For these three automatic metrics, we average the image embeddings over 16 uniformly sampled frames to compute the similarity between video and text embeddings.
UMT [22] is composed of video encoder, text encoder and cross moossl decoder. We adopt the large version of UTM ppe- pre-trained on 25M video-text pairs using four types of losses: Unmasked Token Aligmment (UTA), Video-Text Contrastive (VTC) learning, Video-Text Matching (VTM) and Masked Language Modeling (MLM), with video frames as input. Then it is fine-tuned on MSR-VTT using VTC and VTM, with 12 video frames as input. We utilize the cross-modal decoder's VTM result compute the UTMScore.
To extract elements from video captions and generate questions for Video QA, we utilize the 13B version of Vieuna v1.3 Table shows some examples of extracted elements and generated questions. The full list of elements and questions for the entire FETV dataset and prompts to Vicuna can be found in this link For Otter- Otter-VQA, we employ the video version of Otter which is based on LLaMA7B. We deem the Video LLM's response as positive if begins with "yes" and negative otherwise. Then, the average number of questions that receive a positive answer is defined as the alignment score for a specific video-text pair.
D.2 Video Quality Metrics
As mentioned in Section 4. we collect 300 generated and reference videos of a particular category to compute FID [IT] and F [40]. Specifically, to collect the generated videos, we randomly sample from the 619 prompts in FETV (a prompt can be sampled multiple times), until every major content category has at least 300 sampled prompts, and generate videos using the T2V models. To collect the reference videos, randomly sample from MSR-VTT test set (59,800 prompts) and Web Vid validation set (4,999 prompts), until every major content category has at least 300 sampled prompts, and collect the corresponding videos. In this way, collect 2,055 generated videos (for each T2V model) and 2,328 reference videos. Figure [3] summarizes the distribution of the sampled prompts. To compute FID and FVD for particular category, we sample 300 generated and reference videos that belong to this category, and average over 4 runs with different random seeds.
We adopt clean fid ' [28) implementation of FID and stylegan-v's 35 implementation of FVD.
18

---

Table 5: Examples of extracted elements and generated questions by Vicuna.
Video Description:
people are dancing
Extracted Elements: people, dancing
Generated Questions:
About people: are there people in the video?
About dancing: are the people in the video dancing?
Time lapse moving clouds with blue sky Extracted Elements: time lapse, clouds, moving, blue sky Generated Questions:
Video Description:
About time lapse: is this time lapse?
About clouds: are there clouds the video?
About moving: the clouds moving in the video?
About blue sky: is the video showing a blue sky?
Video Description: cartoon characters driving a plane across a rainbow to the right
Extracted Elements: cartoon characters, driving, plane, rainbow, right
Generated Questions:
About cartoon characters: are there cartoon characters?
About driving: are the cartoon characters driving?
About plane: this video about plane?
About rainbow: is the video showing a rainbow?
About right: the plane driving the right?
Table Inter-human correlation in manual evaluation. Kendall's т. and Spearman's 's are averaged across the correlation between every two humans. - denotes standard deviations. The p-values are less than 0.05.
Static Quality
Temporal Quality
Overall Alignment
Fine-grained Alignment
Kendall's Spcarman's 's p
0.55
0.62=0.066
0.58
0.58=0.03
0.70=#0.0
0.74+t0.06
0.72+0.00
0.72:10.04
Krippendorff's
0.689
0.712
0.674
0.730
E
More Details of Manual Evaluation Setups
E.1
Evaluation Instructions
27 has shown that a specific definition of each rating level can facilitate the inter-human correlation in the manual evaluation of T2I models. Therefore, we carefully design the rating level definitions and provide some examples of ratings as a reference. Figure and Figure [1] show the instructions for manual evaluation of video quality and video-text alignment, respectively.
E.2 Evaluation Interface
Figure/[6jillsttrenn [6]illustrates our manual evaluation interface, which based on the Python tkinter package To alleviate the influence of resolution in the assessment of video quality, we crop all the generated and reference videos 256x256 spatial resolution in the interface. Typically, the reference videos are longer than the generated ones. To prevent the reference videos from being identified based on the duration, we crop them to 4 seconds when evaluating video quality. When evaluating video-text alignment, the reference videos are kept to the original duration to maintain all the content described the text prompts. Considering that the quality of recently rated videos may influence the assessment of the current video (e.g., if the recently rated videos are of poor quality, the evaluator may be inclined lower the standard), we alternately display videos generated by different models.
E.3
Correlation between Human Evaluators
The correlation between human evaluators is shown Table 6 which demonstrates strong inter- human agreement across the four perspectives to evaluate.
19

---

Spatial Major Content
Temporal Major Content
1200
1054
800
767
1000
Sample Count
Sample Count
868
600
800
782
441
461
400
324
345
300
321
300
467
400
200
people
animals
vehicles
plants
artifacts
food
building
scenery
illustrations
actions
kinetic motions
fluid motions
light change
(a) Distribution sampled from FETV generate videos.
Spatial Major Content
Temporal Major Content
2000
1750
1673
1750
Sample Count
Sample Count
1500
1250
1171
1000
750
572
750
653
500
355
356
300
301
500
467
326
250
people
animals vehicles plants
artifacts
food
building
scenery
illustrations
actions
kinetic motions fluid mations
light change
(b) Distribution data sampled from MSR-VTT and WebVid collect reference vidcos.
Figure 13: Sampled data distribution for computing FID and FVD.
F More Experimental Results
F1 Video-Texi Alignment Ranking Results
In Figure of the main paper, we compare automatic and human ranking of the T2V model in terms of video-text alignment over the entire FETV benchmark. The results in specific attribute control categories are shown Figure [7] It can be seen that UMTScore exhibits the strongest correlation with humans in every category, compared with other automatic alignment metrics.
F.2 FID and FVD Ranking Results
In Figure of of the main paper, compare FID/FVD and human ranking of the T2V model in terms of video quality over the entire FETV benchmark. Figure and Figure 19 present the results in specific spatial and temporal categories. We can find that, compared with FID and FVD, FVD-UMT is more consistent with human rankings in almost all categories. However, in some categories (c.g., "Illsstrttion,, "Food & Beverage" and "Plants") all three automatic metries are poorly aligned with humans. For these categories, we still need to rely manual evaluation as the golden standard of video quality.
F.3 The Efffet of Video Number on FID and FVD
As shown in Figure 20 the number videos a substantial impact on values of FID and FVD, which decrease with the increase of video number. However, when the number of videos exceeds 300, FVD-UMT can provide more accurate ranking the T2V models, as compared with FID and FVD. Since generating thousands of vidcos is time-consuming (especially for CogVideo which takes
20

---

Static Quality
Levdl
Level
Generel Instruction
sigh
Despiao Cample
"Meoe 1- bose "o edtand euch sigll torme contorr
ueniknnl anotiit..
oturenant inolece incede
of unciar atormtion
Cores eaim plu video forme chor vodr aich iring lnel with resoone
eep sined belon hames mort stmppee Freax an tre
Lowell
2evel2
Internaion
(a) Static Quality.
Temporal Quality
Cenunal
qudly undly
Leae't
gudtyy grod
(b) Temporal Quality.
Figure 14: Screenshots of instructions for manual evaluation of video quality.
21

---

Overall Video- Text Alignment
Lovil
Genend Indirailion
Deviggin
Felo
Leve2
Eserrden vido herr
ent coliig
lyppny seiiing
(a) Overall Video-Text Alignment.
Fine grained Video- Text Alignment
Levit
General rntruation
copter
capiier Staril lal
oerfinn Fittt senpon Kinkites Coerie siee
octior Your socees
contril
Arabura
Nrg
Indposond
Suant
Labee hig
Touel Conngg bere
Gamers
vntoree Quinty
audting with and oetir Worden strng artice parvor meting them wnterggtton
Ene sorse
Cole
Creect io suppasss the
lyeng and reting quec you semta deeld
The Keep
(b) Fine- sinenerained Video- Vixeo-Text Alignment.
Figure 15: Screenshots of instructions for manual evaluation of video-text alignment.
22

---

Paggett Prodaing pi
11449
Pogpine sing DIFOYL (on0)
1100
Lovds Byte gully Puns with die
Lenit Thenk heicphtaale creal of Pramr Fernitors
gett
Colladig sigeant
collore 1001 FON Mackeing
mtaliyy regat life publing nodar Pioghna (oois
Leain: naso matiees Bil Ho vill naar oPheria Jeiry suuiigg rcanec
de las
Lewil: This vilece mat Mie certoct
Lete LiNEL
Lene
Leann
Lenan
LendO
Figure 16: Screenshots of the manual evaluation interface.
more than 20 minutes to generate one video), we only use 300 videos when computing FID/FVD for each category. More reliable results can be obtained by generating more videos.
F.4 Results on Real-World and Unusual Prompts
Figure 21 compares the models' performance real-world prompts (collected from existing datasets) and our manually-writte unusual prompts. We find that it is more challenging for the models to generate videos following the unusual prompts, while the video quality is less affected.
F55 Qualitative Analysis of Generated Videos
present some examples of the generated videos by the three T2V models. We summarize the key findings in the captions, which are in accordance with the findings from the quantitative analysis in Section In this appendix, we only show screenshots of the videos. For a better understanding the video quality and video-text alignment, especially the temporal categories, please refer to the complete videos.
Figure 24 23
25
26
27
28
29
30
31
F.6 Case study of Video- Text Alignment Metrics
Figure and Figure 32 illustrate examples of videos and video-text alignment score measured by different metrics. We can see that (1) UMTSScore produces the most consistent ranking with humans. (2) The other automatic metics are inclined to assign low alignment score to the ground-truth videos. We conjecture that this is because the ground-truth videos are relatively more complicated than the generated ones (e.g., For the prompt "a horse follows two girls" in Figure 32] the two girls do not appear in all video frames.), which requires deep vision-language understanding ability to evaluate.
G Automatic Categorization Rules
For each category under the "major content" and "attribute control" aspects, we define two types of textual features for matching, i..., WordNet synsets and key phrases/words, which are summarized in Table Then, the prompts that can match any features of a specific category will be classified accordingly. The "prompt complexity" depends the number of non-stop words in prompt, as described in Table
23

---

0.329
0.3
2.48
0.750
CUPScoree
9.315
CLiSScree
BLIPScore
2.47
5.46
ORarAVA
0.725
0.700
3.305
0.835
9.302
0:7
0.450
2.0
0.225
35
6.0
4.5
4.0 Human
e.0
35
4.0
35
45
Human
Human
Human T
Human
Coglidee
MocetSoope--22
TeetZiizeo
ZeriSeap
Gruns-ruus
Results under the "Color" category.
0:30
9.55
CLIPscore 1
0:33
CLiSScor-ft
0.29
0:39
BUPS
Ottec-OA
2.50
2.0
0.27
c28
2.42
0,70
Human
Human
Human
Human 1
Human 1
Results under "Quantity" category.
0.32
5.48
0.73
CLiPScore
0.33
CLIPScore-it
0.30-
BUPScore:
6:46
Ofter vQ*
0.73
0.2
0.60
0.03
0.82
028-
0.5
4.5
Human
Human
Human Teit70srooees
Human
Human
Coglideeo
MacelSopp--P
ZwmScape
Gruns-Thuth
(c) Results under the "Camera View" category.
9.75
CUPSSooee
0.32
CUPScre-tt
0.30-
BLIPScore
Otter-VOAt
0.39
0.29
2:44
2.0
0.65
0.29
3.5
4,8
3.5
4.8
45
Human
Human
Human
Human
Human
Coglidee
Mora
Zwasimps
Grouns-Tuuh
(d) Results under the "Speed" category.
5.40
0.87
CUPS.ore +
0.20
CLisScore--t
0.26
0.30
0.25
ULIPSCoTe
0.46
Otter VQA
0.75
0.75
UNTSccrm
0.29
026
0.27
6:42
0.65
Human
Human 1
Human Trat2iisroows
Human
Human
Coglidee
MocetSoop--T2
Zeaicopp
Gouns-Thurh
Results under the "Motion Direction" category.
028
CUISccree
0.302
0.305
CUPScore-ft
0.27
BLPSScore
2:44
Otter-VOr
0.75
3.60
0.25
0.392
9.50
Human
Human
Human
Human 1
Human
Mocelscoe-T2v
(f) Results under the "Event Order" category.
Figure 17: Automatic and human ranking of T2V models terms of overall video-text alignment across different attribute control categories.
24

---

Ocoaao
Monkcpelie
(a) Results under the "Actions" category.
(b) Results under "Kinetic Motions" category.
10
HaT2R
Haten
(c) Results under the "Fluid Motions" category.
(d) Results under the "Light Change" category.
Figure 18: Automatic and human ranking of T2V models in terms of video quality (averaged static and temporal quality) across different temporal categories.
220
Results under the "People" category.
Results under "Scenery& Natural Objects" category.
Humo Juahaue
Hockbortte
(c) Results under the "Artifacts" category.
(d) Results under the "Vehicles" category.
23
23
12
24
24
(c) Results under the "Animals" category.
(f) Results under the "Buildings & Infrastructure" cate- gory.
1200
Gaylder
Hockhartte
(g) Results under the "Food & Beverage" category.
(h) Results under the "Plants" category.
37
Results under the "Ilustration" category.
Figure 19: Automatic and human ranking of T2V models in terms of video quality (averaged static and temporal quality) across different spatial categories.
25

---

Table 7: Matching features for automatic categorization.
(a) Temporal categories under the "major content" aspect.
WordNet Synsets
Key Phrnses/Words
'tavelly.01, 01", "compete.e.0" 'actx.01'. "manipulae.x.. 'mooe...02", 'chungex001', 'make..000" 'make
Aciios
rotute'. 'move.. 'bounce', sppnn, 'sway,, 'Bythrough', . 'Bl.. "panning" 'drone", 'ran. "walk" 'dive". "zoom", chase" 'samm movemeen" "tall,, 'rise" [sliding video' "sliding camera" "silding that "Forwadl", "backward" "loftward", "rightean", 'upward', "downward"
Kinetic Motions
None
of watern.dll' O1" n..2 atmosphericc phenommmon. deffemmtioonn.02
'lleat, 'ürewark'' 'fire', 'cloud', "clouds' "candle', 'mooke" "wave' 'inthite, "mell' 'shiikk' 'ripple"
bumiin 01'" 'light..01'
"eclipse", 'sunsee', "unrise', "irework'' 'line. 'sunbeam', 'wmm may". 'sunchine", "sunny" "bum", 'shine", "Juminuus' "glow" 'eaplode" 'milky". 'fiabh, spuakle.. 'ncon, reflection. 'bright'. 'cundle"
Light Change
1ight.n022 Tight..0f... "lgght.007"" 'ligh.n099"
(b) Spatial categories under the "major content" aspect.
WordNet Synsets
Key Phrases/Words
People
personnn.011; 'peoplc.n.01"
'he", 'she', "men", 'team'
Animals
'animal.n.01'
None
Vehicles
vehicle.n.011
'drone"
Artifects
aattffatt.n01''
None
Buildings & Infrastructures
'building.n.01 'structure.n.00
'building'. 'cityscape', 'town', "city"
'natural oow__ooojjenn.cc 'body_of,
waternn.1', ooddy_oo_atee...001
Scenery & Natural Object
"gcological_formmaioo... formation.n.011
"mountainous,, 'fire, 'frrework", 'solar eclipse". "water current', 'water drop'. 'cloud', 'desert'
Gammospheric_phenomenoonne 'atmosphere.n.05
Plants
"plant.n.02', 'vgeetation.n.01
None
Food & Beverage
Of0' 'food.n.02'
'mushroom'
"pattern", 'ahstract', 'pattern", 'particle", "radientt. 'loop', 'gaphicc"
Illustrations
'Shape.n.02', 'symbol.a.01'
(c) Categories under the "attribute control" aspect.
WordNet Synsets
Key Phrases/Words
Color
color.n01'.
'white'
"macro shot', 'medium shot", "wide shot', "close up', 'close-up', 'close view", 'close shot", "front view", 'frott facing". 'front facing'. bbackside view". 'backside shot', "side shot', 'top view". "top-down view". 'top down view', "overhend view", "overhead shot', 'bottom view'. 'bottom shot', 'low angle", 'hihh angle", 'acrial', 'drone view", "bird's eye view". 'first person', "first-person. "Ist person', 'third person', third-persont "3rd person"
pprofile pprofile shot', "side
Camera View
None
Quantity
n.0
None
Speed
None
'slow', 'slowly', 'fastly', 'time lapse", 'timelapse', 'time-lapse" 'stop
'ahedd. "anticlockwise', 'away from', "clockwise' 'counterelocwwise" 'downward'; "eastbound", 'northihoudd" 'outhbound "westbound" 'homeward", 'leftwands' righhwards'. 'upward'. , 'left', 'right" 'fowward, 'backward', 'toward'. 'out of". 'approach" 'leave', 'against to". 'lift,, 'opposite direction'
Motion Direction
None
Event Order
None
then'
26

---

600
2500
200
2000-
150
1500
400
100
1000-
500
1000
100
1000
300
800
1000
Videos
Videos
# Videos
CogVideo
ModelScopel 2V
Text2Videe-zero
ZeroScope
Figure 20: The effect of video sample number on FID and FVD. The shaded regions represent the highest and lowest values observed across the four runs. The human evaluation of video quality (average of static and temporal quality) are 3.08, 3.27, 3.62 and 3.93 for Video-zzoo Cog Video, ZeroScope and ModelScope TVV, respectively.
static quality
temporal quality
alignment
Unusual
Figure 21: Manual evaluation results on real-world and unusual prompts.
27

---

Actions
Copvideo
Ton2Video
ModeSSoooeT2
ZeroScope
Ground Trush
Prompt
dinciigg
Spatial Content
people
Prompt
gvils singing
Spottal Content
people
leomnig
to
Spetial Content
people
Promot
roobit: are running around
Soatial Content
onimals
youug playing
Spatial
Content
peopleanimmls
Figure 22: Screenshots of videos generated from prompts of the "Actions" category. Most generated videos contain unnatural elements, including unnatural appearances, inconsistent appearances across frames, unnatural movements and jump in positions.
Kinetic Motions
CosVideo
Tox22Vddo Zoro
ScpeSco
ZeroSoope
Ground Truth
Promot
the cars crove fost
Spatial Content
vehivees
Promot
and sivr motorcycle
Spasial Content
vervilles
Promot
Fiying counter cleowise oround yoche
Sportal Content
whites
Promot
Norse
naрpeninng
onmmb'
Figure 23: Screenshots of videos generated from prompts of the "Kinetic Motions" category. Most generated videos contain unnatural elements, including unnatural appearances, inconsistent appear- ances across frames, unnatural movements and jump in positions.
28

---

Fluid Motions
CogVideo
Ten2Video Zero
Mode T2V
Zero
Ground Trush
Promot
mourtuti stream
adjects
Promot
topse mosing sky
Spartal
scenery
abficcts
Promos
burcing
Spatial
Scenery & netura! otyects
Prampt
Teauting with bens close up view
terstro
Spotial Content
food beverage
Figure 24: Screenshots of videos generated from prompts of the "Fluid Motions" category. Most generated videos are of good quality, except for Text2 Video-Zero, which exhibits poor temporal quality.
Light Change
EngVideo
Test2Video-eera
ModeScooeT2V
ZenoSoope
Groard-Truth
beach
hutterstckk
frewonts are being disploved
Sparial
somerry objects
Promot
butsing
Spatial
scenery & natwrn! atjects
Prompt
Timellpse of moving stars and milky
Spatial
sconery natura!
Content
otjects
Figure 25; Screenshots videos generated from prompts of the "Light Change" category. Most generated videos are of good quality, except for Text2Video-Zera which exhibits poor temporal quality.
29

---

Color
CopVdeo
deo-
ModeB
ZeroScope
Prompt
someone shivig orange sports
Nutters.er
Promot
mon red slvee whheety motoreycle
Prompt
blsamiing
topae blue
burigrmund.
hhutters
Figure 26: Screenshots videos generated from prompts of the "Color" category. All T2V models control color in most cases.
Camera View
LogVideo
lest/Video-Zern
ModelSoope T2V
Zeroboone
Ground Trath
ved formudo
ttock
Tracitional plode wicddn tackground. too vee
Prompt
Aerol wew of olouds of smoke and sheamh caming from
Prompt
man is byping on a wirkless leyboord clsse up.
Figure 27: Screenshots videos generated from prompts of the "Camera View" category. All T2V models can control camera view in most cases.
30

---

Quantity
Cogooown
Tec2VVden-Jero
ModelscopeTas
ZaneSopee
Prompt
Too giik HirGw stones knto wather slow mosion
Prompt
chrze bays are eatiig burger
Prompt
zinoin
meatum aoe skilet
Prompt
two men stanaing nowr catching
Figure 28: Screenshots of videos generated from prompts of the "Quantity" category. Mod- eVV and Text2Video-Zer can control the quantity of objects some extent, while Cog fails in most cases.
Speed
CogVicuo
Text2V Kien
Scope
Prompt
Two giis throw stones into the
Prompt
illrn miloned car ractng very jest lerwing other venicler
Prompt
(ying makive
Prompt
baby running very fest.
Figure 29: Screenshots of videos generated from prompts of the "Speed" category. ModelScopeT2y control both fast and slow speeds some extent. CogVider Video can generate some videos with "slow motion" while struggling with fast speed. For Texe2Vieo--Zero, speed can only be reflected via static information g..g.. the posture of the baby in the last case).
31

---

Motion Direction
Coggideo
Modee EVV
ZeroScape
Ground- Tranh
Prompt
Flying counter
lorge yargeyache yache
wide ahot of beach comerg moves forword
shott ere
Prampt
beauetfil becot.... the comera slowly eiting diwnwarrdr.
utterstock
Prompt
gorden white plone, th cornera zouma inco the solod'
Figure 30: Screenshots of videos generated from prompts of the "Motion Direction" category. ModelScopeT2V and ZeroScope can control motion direction to some extent, while CogVideo and Text2 Zero fail in most cases.
Event Order
Coggiden
dest2Vidoo-Zer
MudelScope2V
ZeroScope
Ground
Prompt
guy pototoes and tnen odds them to bowl heary cream
Prompt
couple of people fightin and them making hugging
Prompt
womoo
walking and then sitting
Figure 31: Screenshots of videos generated from prompts of the "Event Order" category. All T2V models cannot generate videos according the given event order.
32

---

somzone is walking fast along the beach with many benches
Ground Truth
Human
1.044hh)
2.67/2nd
UMTScore
0.375
(3rd)
269(2no)
3.988/10 Lst)
BLIPScore
0.390
0.414[3rd]
0.436iLst)
0.435/2nd)
CUPSscor-ftt
0.258
(.281/3rd)
0.287/2nc)
0.290(1xx)
CLiPScore
28212st1
2.253/3rd
0.263 3rd]
0. 2n268(2dd)
Otter-VOA
1.80
0.80
0.60
0.72(Src)
Prompt
man present women with a rose and then she kisses him on the cheek and they go shopping
Generate
Ground Truth
Human
2.33/4m))
2.67/2nd)
5.01 2as)
(.872
1.947)2rd)
4.343/1st)
BLIPScore
4.404)4m
0.448(1s1)
0.443/2nd]
CUPScore-ft (r2r)))
0.283/2rd)
CLiPScore
(.291/4th)
0.322)2nd)
0.335/1st)
Otter-VOA
0.825(2nc)
0.714:4th
0.85711st
(.829|2nd)
Prompt
girls
Ground Trush
Human
2r0(3rd)
2 67(2nd)
2.67/2nd)
5.0
UMTScore
(1.085/11t)
3vd
3.90612rd)
4.234/1st
BUPScore
(3398411h
0.41013rd)
k.489)1
0.434/2nc)
CLIPScore-ft
0.786(151]
0.258/3rd)
0.274/2ndi
0.322/1x))
Otter-VOA
1.011st)
10t
1.0 s1)
1 1.0((st)
Figure 32: Video examples and alignment scores measured by different metrics. The left three videos generated and the rightmost video is the ground-truth.
33

---

H
Licensing, Hosting and Maintenance Plan
Author Statement.
We bear all responsibilities for the licensing, distribution, and maintenance of
dataset.
License. FETV is under CC-BY 4.0 license.
Hosting, FETV can viewed and downloaded on GitHub at httpttpss//gthuu.co///////////// FETV on Huggingface at thtttss//huggiiii http://ac..c/aaatt//e/aaaaaaa co//attasets/1y27//FEt/ and will be pre-
served for a long time. The dataset is in the JSON file format.
Metadata.
Metadata can be found at http::///hanggnaaaeee.oorr/ttsts
I
Datasheet
I.1 Motivation
For what purpose was the dataset created?
Answer: FETV is created to facilitate fine-grained evaluation in the task of open-domain text- to-video generation. Compared with previous benchmarks for this task, FETV provides more comprchensive view of T2V models' capabilities from multiple aspects.
Who created the dataset (e.g,, which team, research group) and behalf of which entity (e-g., company, institution, organization)?
Answer:
This evaluation dataset is created by Yuanxin Liu Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen and Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab).
Who funded the creation of the dataset?
Answer: Peking University and Huawei Technologies Co., Ltd.
1,2
Composition
What do the instances that comprise the dataset represent? (e.g., documents, photos, people, countries)
Answer:
Our data is provided in JSON format. Each data instance consists of (1) text prompt, (2) the categorization labels under three aspects, (3) the source the prompt (collected from MSR-VTT and Web Vid manually written by us), (4) ID and (5) URL link of the reference video.
How many instances are there in total (of each type, if appropriate)?
Answer: In total, we collect 619 instances. The data distribution over different categories can be found in Figure and Figure of the main paper.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
Answer: part of the text prompts in FETV collected from existing datasets of open-domain text-video pairs (i.e., MSR-VTT and WebVid). When collecting prompts, we ensure that each category our proposed categorization system has sufficient prompts.
Is there label or target associated with each instance?
Answer: Each prompt collected from MSR-VTI VTT and WebVid is associated with reference video, which is provided in the form of an URL link.
Is any information missing from individual instances?
Answer: For the manually written unusual prompts, FETV does not provide the URL link to reference videos.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
Answer: The text prompts in FETV belong to different categories, which can be seen from the categorization labels.
Are there recommended data splits (e.g., training, development/validation testing)?
34

---

Answer: No. The entire FETV benchmark is intended for evaluation.
Are there any errors, sources noise, or redundancies in the dataset?
Answer:
No.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
Answer: The reference videos are provided as URL links because we do not own the copyright to the videos.
Does the dataset contain data that might be considered confidential?
Answer: No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
Answer:
No.
L,3
Collection Process
The data collection process is described in Section 3.2 of the main paper and Appendix G
1.4
Uses
Has the dataset been used for any tasks already?
Yes. We have used the FETV benchmark evaluate three existing text-to-video generation models and diagnose the reliability of automatic video quality and video-text alignment metrics. Please Section and Section of the main paper for details.
Answer:
What (other) tasks could the dataset be used for?
Answer: FETV is mainly designed to evaluate open-domain text-to-video generation models, but it also be extended to test the reliability of automatic evaluation metrics, by computing their correlation between human judgements (as shown in Section 4).
Is there repository that links to any all papers systems that use the dataset?
Answer: No.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/cleaned/labelee that might impact future uses?
Answer: the first stage of our data collection process, some prompts that actually belong to particular category might be filtered out by the automatic matching rules (introduced in Appendix G which potentially impacts the language diversity of the prompts.
Are there tasks for which the dataset should not be used?
Answer: FETV mainly consists of sentence-level text prompts, and thus it is not suitable for evaluating video generation based paragraph-level texts.
I,5 Distribution
Will the dataset distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
Answer: Yes. The benchmark publicly available on the Internet.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?
Answer: The benchmark is available on GitHub at oo/Cattasets/lyy9//EEi
or on
Huggingface at /bttps:/huugingfffae
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms use (ToU)?
Answer: CC-By 4.0.
35

---

Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
Answer:
No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
Answer:
No.
I.6
Maintenance
Who will be supporingghustinnnaaaattiingg the dataset?
Answer: The authors will be supporting, hosting, and maintaining the dataset.
How can the owner/curraorr/mannge of the dataset be contacted (e.g,, email address)?
Answer: Please contact Yuanxin Liu (liuyuanxin @ wtu. wkw.pku..eu.c)..
Is there an erratum?
Answer: No. We will make announcements if there are any.
Will the dataset updated (e.g., to correct labeling errors, add new instances, delete instances)?
Answer: Yes. We will post new update in httth:::///hheeeee httppp://wuggiggfffe..o///////// http://aww...o.taaaaaaa/////////
and https: thub. com 11tmm//Exxx there any.
If the dataset relates people, are there applicable limits on the retention of the data associated with the instances (e-g,, were individuals in question told that their data would be retained for fixed period of time and then deleted)?
Answer: People may appear in the reference videos. People may contact us to exclude specific data instances if they appear in the reference videos.
Will older versions of the dataset continue to be supprrtedhnstee/miinniiie??
Answer: Yes. Old versions will also be hosted in httplp/////uuuggggaaeeeo/ttaaats FETV and htttps:://gtthubbbbb com wwt.:/c//me111eeeee1 /FETV
If others want to extend/augmmen//hiil on/contribute to the dataset, is there a mechanism for them to do so?
Answer: Yes. Others can extend FETV to collect more prompts for evaluation, based on our
multi-aspect categorization system (introduced in Section 3.1 of the main paper).
36


Reviews:
[{"review_id": "vq4gmwnz7b", "reviewer": "Reviewer_HZhg", "summary": "", "questions": "", "limitations": "Although they reveal the reliability problem of automatic T2V evaluation metrics and show some potential solutions for improvement, they do not present a specific automatic metric that can reliably reflect human judgments.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The work proposes a novel benchmark for fine-grained evaluation of T2V generation in a comprehensive view. \n\n2. Their research reveals that the existing metrics correlate poorly with human judgments, giving other researchers evidence to improve T2V generation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TZUVmUUoQz", "reviewer": "Reviewer_gcDZ", "summary": "", "questions": "", "limitations": "This paper reveals the mismatch between existing automatic metrics and human preferences in the evaluation of T2V models. However, it still relies on manual evaluation. As a benchmark paper, it is important to devise and implement a suitable automatic metric, thereby alleviating the dependency on human annotation and making the evaluation process more efficient and cost-effective. \n\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper introduces a new benchmark, thoughtfully crafted to evaluate open-domain text-to-video generation models. Notably, it goes beyond previous benchmarks by incorporating the fine-grained temporal aspect of video generation, giving a holistic view of T2V models' capabilities. The experiments conducted are rigorous, and the analysis provided offers valuable insights, contributing to the advancement of T2V research.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "s7HrERx4bF", "reviewer": "Reviewer_2CxJ", "summary": "", "questions": "", "limitations": "The paper has discussed corresponding limitations", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Collected 619 videos and corresponding prompts from the MSR-VTT test set and WebVid, and provided dataset distribution statistics, including prompt complexity distribution statistics and major content distribution statistics.\n\n2. The work includes a substantial amount of experimental analysis, such as manual evaluation of static and temporal video quality (Figure 5), and metric evaluation (Table 3). It also provides some insights, stating that although the automatic metrics are positively correlated with manual evaluations, the overall correlation degree is weak, being lower than 50% of inter-human correlations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0ptvo3VXFJ", "reviewer": "Reviewer_5H2c", "summary": "", "questions": "", "limitations": "As mentioned above, there are two major limitations from my point of view, which are the failure of finding an **automatic evaluation** and the insufficient discussion on **temporal consistency**. I have justified my concerns in the Additional Feedback section in detail.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- S1. The proposed FETV includes evaluation from various aspects, which is more comprehensive than other related benchmarks.\n- S2. The authors succeed in pointing out the limitation of automatic evaluation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Three of the four reviewers support borderline accept (score 6). One reviewer 2CxJ scores marginally below acceptance (5). The main concerns of 2CxJ are (1) the uniqueness of the dataset and (2) weak contributions especially on manual grading score vs. CLIPScore and BLIPScore, and on lacking an automatic metric (also pointed out by other reviewers). The authors have responded extensively. Unfortunately reviewer 2Cx didn't follow-up on the authors comprehensive responses to raise any remaining issues. I consider that the two issues of existing text-to-video datasets the authors identified are important and this dataset is timely. It does offer two critical unique perspectives in both temporal-aware and multi-aspect as compared to existing datasets. Moreover, the authors' explanation on the value of manual grading score is persuasive, and the added automatic metric for evaluation in the revised paper is credible. On balance I recommend accept.
Decision: Accept (Poster)