SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the basic version of CLIPScore, we adopt the implementation from [18], with ViT-B/32 [7] as the visual encoder backbone. For BLIPScore, we utilize the BLIP model $^{3}$  [21], with ViT-B/16 as the backbone, fine-tuned on the COCO dataset [23] for image-text retrieval. For CLIPScore-ft, we fine-tune the ViT-B/32 CLIP model for video-text retrieval on the 7k training split of MSR-VTT [8], which has no overlap with the reference videos in FETV. The implementation and hyper-parameters settings follow CLIP4CLIP [24]. For these three automatic metrics, we average the image embeddings over 16 uniformly sampled frames to compute the similarity between video and text embeddings.
UMT [22] is composed of a video encoder, a text encoder and a cross-modal decoder. We adopt the large version of UTM pre-trained on 25M video-text pairs using four types of losses: Unmasked Token Alignment (UTA), Video-Text Contrastive (VTC) learning, Video-Text Matching (VTM) and Masked Language Modeling (MLM), with 4 video frames as input. Then it is fine-tuned on MSR-VTT using VTC and VTM, with 12 video frames as input. We utilize the cross-modal decoder's VTM result to compute the UTMScore.
To extract elements from video captions and generate questions for Video QA, we utilize the 13B version of Vicuna v1.3. Table 5 shows some examples of extracted elements and generated questions. The full list of elements and questions for the entire FETV dataset and prompts to Vicuna can be found in this link. For Otter-VQA, we employ the video version of Otter, which is based on LLaMA7B. We deem the Video LLM's response as positive if it begins with "yes" and negative otherwise. Then, the average number of questions that receive a positive answer is defined as the alignment score for a specific video-text pair.
D.2 Video Quality Metrics
As mentioned in Section 4.2, we collect 300 generated and reference videos of a particular category to compute FID [11] and  $\mathrm{FVD^4}$  [40]. Specifically, to collect the generated videos, we randomly sample from the 619 prompts in FETV (a prompt can be sampled multiple times), until every major content category has at least 300 sampled prompts, and generate videos using the T2V models. To collect the reference videos, we randomly sample from MSR-VTT test set (59,800 prompts) and WebVid validation set (4,999 prompts), until every major content category has at least 300 sampled prompts, and collect the corresponding videos. In this way, we collect 2,055 generated videos (for each T2V model) and 2,328 reference videos. Figure 13 summarizes the distribution of the sampled prompts. To compute FID and FVD for a particular category, we sample 300 generated and reference videos that belong to this category, and average over 4 runs with different random seeds.
<sup>3</sup>https://github.com/salesforce/BLIP
4We adopt clean-fid's [28] implementation of FID and stylegan-v's [35] implementation of FVD.
Video Description: people are dancing
Extracted Elements: people, dancing
Generated Questions:
- About people: are there people in the video?
- About dancing: are the people in the video dancing?
Video Description: Time lapse moving clouds with blue sky
Extracted Elements: time lapse, clouds, moving, blue sky  
Generated Questions:
- About time lapse: is this a time lapse?
- About clouds: are there clouds in the video?
- About moving: are the clouds moving in the video?
- About blue sky: is the video showing a blue sky?
Video Description: cartoon characters driving a plane across a rainbow to the right
Extracted Elements: cartoon characters, driving, plane, rainbow, right
- About cartoon characters: are there cartoon characters?
- About driving: are the cartoon characters driving?
- About plane: is this video about a plane?
- About rainbow: is the video showing a rainbow?
- About right: is the plane driving to the right?
Table 6: Inter-human correlation in manual evaluation. Kendall's  $\tau_{c}$  and Spearman's  $\rho$  are averaged across the correlation between every two humans.  $\pm$  denotes standard deviations. The p-values are less than 0.05.
Static Quality	Temporal Quality	Overall Alignment	Fine-grained Alignment
Kendall's τc	0.55±0.04	0.62±0.06	0.58±0.03	0.58±0.03
Spearman's ρ	0.70±0.05	0.74±0.06	0.72±0.04	0.72±0.04
Krippendorff's α	0.689	0.712	0.674	0.730
E More Details of Manual Evaluation Setups
E.1 Evaluation Instructions
[27] has shown that a specific definition of each rating level can facilitate the inter-human correlation in the manual evaluation of T2I models. Therefore, we carefully design the rating level definitions and provide some examples of ratings as a reference. Figure 14 and Figure 15 show the instructions for manual evaluation of video quality and video-text alignment, respectively.
E.2 Evaluation Interface
Figure 16 illustrates our manual evaluation interface, which is based on the Python tkinter package. To alleviate the influence of resolution in the assessment of video quality, we crop all the generated and reference videos to  $256 \times 256$  spatial resolution in the interface. Typically, the reference videos are longer than the generated ones. To prevent the reference videos from being identified based on the duration, we crop them to 4 seconds when evaluating video quality. When evaluating video-text alignment, the reference videos are kept to the original duration to maintain all the content described in the text prompts. Considering that the quality of recently rated videos may influence the assessment of the current video (e.g., if the recently rated videos are of poor quality, the evaluator may be inclined to lower the standard), we alternately display videos generated by different models.
E.3 Correlation between Human Evaluators
The correlation between human evaluators is shown in Table 6, which demonstrates strong interhuman agreement across the four perspectives to evaluate.
Table 5: Examples of extracted elements and generated questions by Vicuna.
(a) Distribution of data sampled from FETV to generate videos.
(b) Distribution of data sampled from MSR-VTT and WebVid to collect reference videos.
Figure 13: Sampled data distribution for computing FID and FVD.
F More Experimental Results
F.1 Video-Text Alignment Ranking Results
In Figure 8 of the main paper, we compare automatic and human ranking of the T2V model in terms of video-text alignment over the entire FETV benchmark. The results in specific attribute control categories are shown in Figure 17. It can be seen that UMTScore exhibits the strongest correlation with humans in every category, as compared with other automatic alignment metrics.
F.2 FID and FVD Ranking Results
In Figure 10 of the main paper, we compare FID/FVD and human ranking of the T2V model in terms of video quality over the entire FETV benchmark. Figure 18 and Figure 19 present the results in specific spatial and temporal categories. We can find that, compared with FID and FVD, FVD-UMT is more consistent with human rankings in almost all categories. However, in some categories (e.g., "Illustration", "Food & Beverage" and "Plants") all three automatic metrics are poorly aligned with humans. For these categories, we still need to rely on manual evaluation as the golden standard of video quality.
F.3 The Effect of Video Number on FID and FVD
As shown in Figure 20, the number of videos has a substantial impact on the values of FID and FVD, which decrease with the increase of video number. However, when the number of videos exceeds 300, FVD-UMT can provide more accurate ranking of the T2V models, as compared with FID and FVD. Since generating thousands of videos is time-consuming (especially for CogVideo which takes
Static Quality
General Instruction
- Please rate the videos from 1-5 based on "to which extent each single frame contains unnatural artefacts".
- Types of unnatural artefacts include
8.unnatural appearance (e.g., in shape and texture)
a. lack of or unclear visual information
- Some example video frames are shown under each rating level, with the reasons for rating explained below the frames (for most examples). Please use them as a reference.
Description: The single frames are completely unnatural
Examples
completely unnatural appearance
no meaningful visual information
Description: The single frames contain very unnatural artefacts.
Examples
very unnatural appearance
veryunnatur appearance
very unclear visual information.
there are unnatural artefacts in a small portion of frames.
(a) Static Quality.
on: The temporal quality is generally acceptable, but with a few unnatural frames.
.
the appearance of the person in the room. The patient's position is not jump in his positions is not moving.
the appearance of the person in the room. The person is jumping in his position is not visible.
Description: The temporal quality is good, only with minor unnatural frame transitions.
the movement of the cat is slightly unnatural
(b) Temporal Quality.
$\therefore m = \frac{3}{11}$  ;
the object in the woman's hand is slightly inconsistent
the appearance of the background snow is slightly inconsistent before and after the penguin passed
- Description: There are no recognizable unnatural frame transitions.
- Examples
completely inconsistent appearance across frames It's very unnatural, spontaneously.
completely inconsistent appearance across frames
completely unnatural deformation
severe jump in position
Description: The frame transitions are very unnatural.
very inconsistent objects & jump in position
very inconsistent appearance across frames
Figure 14: Screenshots of instructions for manual evaluation of video quality.
very incors
very unimr movements
- Overall Video-Text Alignment
- Please rate a video from 1-5 based on its "overall alignment with the caption".
- Some example videos are shown under each rating level, with the reasons for rating explained below the videos (for most examples). Please use them as a reference.
Description: This video is completely unrelated to the text.
caption: a rocket goes off into the wide blue sky over the tree.
caption: Shaving facial skin macro closeup of man point of view camera movement
- Description: This video barely matches the text, at least half of the elements are missing, incorrect or very unclear.
Examples
caption: someone heats a pot of water and melts chocolate
"chocolate" is very unclear and "with boiling water" is missing.
caption: rabbits ar running around
(a) Overall Video-Text Alignment.
this is a simple prompt, but the action is incorrect.
caption: Couple of people.  
A couple of people.  
up by fugging each other.
this prompt describes two cases of a person who is hugging with no details, but the other is a woman.
Description: This video basically matches the text, but there are a few elements that are missing, incorrect or unclear.
Examples
caption: baby kittens are keeping
It is unclear whether the cat in the video is a "baby kitten".
caption: four airplanes flying
caption: a young man in sunglasses walks down a street.
the details of the man (young and sunglasses) and the scene (i.e., hill) are correct, but "towards a lake is missing".
- Description: This video matches the text very well, with minor differences in terms of missing, incorrect or unclear elements.
Examples
Caption: River in
mountains.
It's slightly unclear when the water in the video is mountain river'or lake.
caption: Profile view of young handsome bearded indian businessman looking back
Supplementary vericator most of the details are correct except for missing "looking back"
caption: a horse race is happening
it is slightly unclear whether it would be better suited to just horse running in the field.
- Description: This video perfectly matches the text.
- Examples
caption: rabbits are running around
Fine-grained Video-Text Alignment
Please rate a video from 1-3 based on its 'alignment with the caption in a specific way' (e.g., 'I like this')
Color, Quantity, Camera View, Speed, Motion Direction, Event Order
- Some example videos are shown under each rating level, with the reasons for rating explained below the videos (for most examples). Please use them as a reference.
This video does not match the text in this attribute to control.
caption: Young canine in a beautiful lake with close up on the sea. Speed: 100 mph. Attribute to control: Camera view
The video is in correct in "up on paddle".
caption couple of people fighting and then making up by hugging each other to control the order.
Figure 15: Screenshots of instructions for manual evaluation of video-text alignment.
the event of, "fighting" is missing
caption: Close up footage rain drops falling off the ground with green nature mountains background.
- control (green nature mountain) is missing.
- caption: a white truck drives very fast through the town.
(b) Fine-grained Video-Text Alignment.
the speed is not "fast"
caption: four airplanes; flying   
Attribute to control:   
Quantity
caption: a camera moves toward and zooms in on a camera.  
Attribute to control Motion Direction
in" is missing
- This video partly matches the text in this attribute to control.
- Examples
caption: Scarlet ibis bird drinking water in the river
the color of the ibis is not similar to "scarflet"
caption: Close up of it is unclear whether the video is a "first person view"
pudding with nuts and fruits in slow motion. Attribute to control: Speed
"zoom in" is correct but "moves into view from the top" and "disappears to the top" are incorrect.
- This video matches the text in this attribute to control.
- Examples
caption: Driving forward under arch of leaf trees canopy during autumn season at sunrise over sealed road. Dimension to control:
caption: two women talking on the camera Dimension to control Quantity
caption: the men race thehorses really fast in thegrass
Dimension to control: Speed
caption: aerial view of surfers on longboards with oars sailing in bay. 2015 successful summer survey
Dimension to control: Camera view
caption: a red-headed woman is shown walking and then sitting
Dimension b
Event Order
caption: Red hibiscus flower blooming in time-lapse on a blue background. Densification of control
Color
Figure 16: Screenshots of the manual evaluation interface.
more than 20 minutes to generate one video), we only use 300 videos when computing FID/FVD for each category. More reliable results can be obtained by generating more videos.
F.4 Results on Real-World and Unusual Prompts
Figure 21 compares the models' performance on real-world prompts (collected from existing datasets) and our manually-written unusual prompts. We find that it is more challenging for the models to generate videos following the unusual prompts, while the video quality is less affected.
F.5 Qualitative Analysis of Generated Videos
Figure 22, 24, 23, 25, 26, 27, 28, 29, 30, 31 present some examples of the generated videos by the three T2V models. We summarize the key findings in the captions, which are in accordance with the findings from the quantitative analysis in Section 4.3. In this appendix, we only show screenshots of the videos. For a better understanding of the video quality and video-text alignment, especially in the temporal categories, please refer to the complete videos.
F.6 Case study of Video-Text Alignment Metrics
Figure 9 and Figure 32 illustrate examples of videos and video-text alignment score measured by different metrics. We can see that (1) UMTScore produces the most consistent ranking with humans. (2) The other automatic metrics are inclined to assign low alignment score to the ground-truth videos. We conjecture that this is because the ground-truth videos are relatively more complicated than the generated ones (e.g., For the prompt "a horse follows two girls" in Figure 32, the two girls do not appear in all video frames.), which requires deep vision-language understanding ability to evaluate.
G Automatic Categorization Rules
For each category under the "major content" and "attribute control" aspects, we define two types of textual features for matching, i.e., WordNet synsets and key phrases/words, which are summarized in Table 7. Then, the prompts that can match any features of a specific category will be classified accordingly. The "prompt complexity" depends on the number of non-stop words in a prompt, as described in Table 3.
(a) Results under the "Color" category.
(b) Results under the "Quantity" category.
(c) Results under the "Camera View" category.
(d) Results under the "Speed" category.
(e) Results under the "Motion Direction" category.
Figure 17: Automatic and human ranking of T2V models in terms of overall video-text alignment across different attribute control categories.
(f) Results under the "Event Order" category.
(a) Results under the "Actions" category.
(b) Results under the "Kinetic Motions" category.
(c) Results under the "Fluid Motions" category.
(d) Results under the "Light Change" category.
Figure 18: Automatic and human ranking of T2V models in terms of video quality (averaged static and temporal quality) across different temporal categories.
(a) Results under the "People" category.
(b) Results under "Scenery&Natural Objects" category.
(c) Results under the "Artifacts" category.
(d) Results under the "Vehicles" category.
(e) Results under the "Animals" category.
(f) Results under the "Buildings & Infrastructure" category.
(g) Results under the "Food & Beverage" category.
(h) Results under the "Plants" category.
(i) Results under the "Illustration" category.
Figure 19: Automatic and human ranking of T2V models in terms of video quality (averaged static and temporal quality) across different spatial categories.
Table 7: Matching features for automatic categorization.
(a) Temporal categories under the "major content" aspect.
WordNet Synsets	Key Phrases/Words
Actions	‘travel.v.01’, ‘compete.v.01’, ‘act.v.01’, ‘manipulate.v.02’, ‘move.v.03’, ‘move.v.02’, ‘change.v.01’, ‘make.v.03’, ‘make.v.01’	None
Kinetic Motions	None	‘rotate’, ‘move’, ‘bounce’, ‘spin’, ‘sway’, ‘flythrough’, ‘fly’, ‘panning’, ‘drone’, ‘run’, ‘walk’, ‘drive’, ‘zoom’, ‘chase’, ‘swim’, ‘movement’, ‘fall’, ‘rise’, ‘sliding video’, ‘sliding camera’, ‘sliding shot’, ‘forward’, ‘backward’, ‘leftward’, ‘rightward’, ‘upward’, ‘downward’
Fluid Motions	‘body_of_water.n.01’, ‘fluid.n.01’, ‘fluid.n.02’, ‘atmospheric_phenomenon.n.01’, ‘deformation.n.02’	‘fountain’, ‘float’, ‘firework’, ‘fire’, ‘cloud’, ‘clouds’, ‘candle’, ‘smoke’, ‘wave’, ‘inflate’, ‘melt’, ‘shrink’, ‘ripple’
Light Change	‘burning.n.01’, ‘light.n.01’, ‘light.n.02’, ‘light.n.04’, ‘light.n.07’, ‘light.n.09’	‘eclipse’, ‘sunset’, ‘sunrise’, ‘firework’, ‘fire’, ‘sunbeam’, ‘sun ray’, ‘sunshine’, ‘sunny’, ‘burn’, ‘shine’, ‘luminous’, ‘glow’, ‘explode’, ‘milky’, ‘galaxy’, ‘flash’, ‘sparkle’, ‘neon’, ‘reflection’, ‘bright’, ‘candle’
(b) Spatial categories under the "major content" aspect.
WordNet Synsets	Key Phrases/Words
People	‘person.n.01’, ‘people.n.01’	‘he’, ‘she’, ‘men’, ‘team’
Animals	‘animal.n.01’	None
Vehicles	‘vehicle.n.01’	‘drone’
Artifacts	‘artifact.n.01’	None
Buildings & Infrastructures	‘building.n.01’, ‘structure.n.01’	‘building’, ‘cityscape’, ‘town’, ‘city’
Scenery & Natural Object	‘natural_object.n.01’, ‘body_of_water.n.01’, ‘geologicalFormation.n.01’, ‘atmospheric Phenomenon.n.01’, ‘atmosphere.n.05’	‘mountainous’, ‘fire’, ‘firework’, ‘solar eclipse’, ‘water current’, ‘water drop’, ‘cloud’, ‘desert’
Plants	‘plant.n.02’, ‘vegetation.n.01’	None
Food & Beverage	‘food.n.01’, ‘food.n.02’	‘mushroom’
Illustrations	‘shape.n.02’, ‘symbol.n.01’	‘pattern’, ‘abstract’, ‘pattern’, ‘particle’, ‘gradient’, ‘loop’, ‘graphic’
(c) Categories under the "attribute control" aspect.
WordNet Synsets	Key Phrases/Words
Color	‘color.n.01’,	‘white’
Camera View	None	‘macro shot’, ‘medium shot’, ‘wide shot’, ‘close up’, ‘close-up’, ‘close view’, ‘close shot’, ‘front view’, ‘front-facing’, ‘front facing’, ‘backside view’, ‘backside shot’, ‘profile view’, ‘profile shot’, ‘side view’, ‘side shot’, ‘top view’, ‘top-down view’, ‘top down view’, ‘overhead view’, ‘overhead shot’, ‘bottom view’, ‘bottom shot’, ‘low angle’, ‘high angle’, ‘aerial’, ‘drone view’, “bird's eye view”, “first person”, “first-person”, ‘1st person’, ‘third person’, ‘third-person’, ‘3rd person’
Quantity	‘integer.n.01’	None
Speed	None	‘slow’, ‘fast’, ‘slowly’, ‘fastly’, ‘time lapse’, ‘timelapse’, ‘time-lapse’, ‘stop motion’
Motion Direction	None	‘ahead’, ‘anticlockwise’, ‘away from’, ‘clockwise’, ‘counterclockwise’, ‘downward’, ‘eastbound’, ‘northbound’, ‘southbound’, ‘westbound’, ‘homeward’, ‘leftwards’, ‘rightwards’, ‘upward’, ‘left’, ‘right’, ‘forward’, ‘backward’, ‘toward’, ‘out of’, ‘approach’, ‘leave’, ‘against to’, ‘lift’, ‘opposite direction’
Event Order	None	‘and then’
Figure 20: The effect of video sample number on FID and FVD. The shaded regions represent the highest and lowest values observed across the four runs. The human evaluation of video quality (average of static and temporal quality) are 3.08, 3.27, 3.62 and 3.93 for Text2Video-zero, CogVideo, ZeroScope and ModelScopeT2V, respectively.
Figure 21: Manual evaluation results on real-world and unusual prompts.
Figure 22: Screenshots of videos generated from prompts of the "Actions" category. Most generated videos contain unnatural elements, including unnatural appearances, inconsistent appearances across frames, unnatural movements and jump in positions.
Figure 23: Screenshots of videos generated from prompts of the "Kinetic Motions" category. Most generated videos contain unnatural elements, including unnatural appearances, inconsistent appearances across frames, unnatural movements and jump in positions.
Figure 24: Screenshots of videos generated from prompts of the "Fluid Motions" category. Most generated videos are of good quality, except for Text2Video-Zero, which exhibits poor temporal quality.
Figure 25: Screenshots of videos generated from prompts of the "Light Change" category. Most generated videos are of good quality, except for Text2Video-Zero, which exhibits poor temporal quality.
Figure 26: Screenshots of videos generated from prompts of the "Color" category. All T2V models can control color in most cases.
Figure 27: Screenshots of videos generated from prompts of the "Camera View" category. All T2V models can control camera view in most cases.
Two girls throw stones into the water, slow motion.
CogVideo
ModelScopeT2V
ZeroScope
three boys are eating burger
a man cooking two sirloin steaks in a medium size skillet
two men standing by a river catching fish
Figure 28: Screenshots of videos generated from prompts of the "Quantity" category. ModelScopeT2V and Text2Video-Zero can control the quantity of objects to some extent, while CogVideo fails in most cases.
Figure 29: Screenshots of videos generated from prompts of the "Speed" category. ModelScopeT2V can control both fast and slow speeds to some extent. CogVideo can generate some videos with "slow motion" while struggling with fast speed. For Text2Video-Zero, speed can only be reflected via static information (e.g., the posture of the baby in the last case).
Figure 30: Screenshots of videos generated from prompts of the "Motion Direction" category. ModelScopeT2V and ZeroScope can control motion direction to some extent, while CogVideo and Text2Video-Zero fail in most cases.
Figure 31: Screenshots of videos generated from prompts of the "Event Order" category. All T2V models cannot generate videos according to the given event order.
Figure 32: Video examples and alignment scores measured by different metrics. The left three videos are generated and the rightmost video is the ground-truth.
H Licensing, Hosting and Maintenance Plan
Author Statement. We bear all responsibilities for the licensing, distribution, and maintenance of our dataset.
License. FETV is under CC-BY 4.0 license.
Hosting. FETV can be viewed and downloaded on GitHub at https://github.com/1lyx97/FETV or on Huggingface at https://huggingface.co/datasets/1lyx97/FETV, and will be preserved for a long time. The dataset is in the JSON file format.
Metadata. Metadata can be found at https://huggingface.co/datasets/lyx97/FETV.
I Datasheet
I.1 Motivation
Answer: FETV is created to facilitate fine-grained evaluation in the task of open-domain text-to-video generation. Compared with previous benchmarks for this task, FETV provides a more comprehensive view of T2V models' capabilities from multiple aspects.