SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We only compute automatic metrics on the generated videos, disregarding the conditioning ground-truth video (action execution and story continuation tasks). For story generation, the frames generated from the 'background' descriptions are omitted, and we do not report SIM as it is ill-defined here. We release our evaluation code to promote reproducible and comparable research on STORYBENCH.
4 Training Data Challenge
In addition to the challenges of video generation discussed in Section 3, the lack of large amounts of high-quality data is a major bottleneck to train strong text-to-video models. Manually collecting annotations for videos is both expensive and time-consuming, especially in story-like format.
Given the major role that data plays in training state-of-the-art generative models, and the impact that careful pre-processing and synthetic data can have, we explicitly define the challenge of training data curation to improve performance in STORYBENCH.
We provide a first approach in this work, which we will show to be beneficial in Section 6. Specifically, we define an automatic pipeline to transform the original VidLN captions for Oops and UVO into multiple sentences, each approximately describing a single action, and determine their timestamps.
To create an original VidLN annotation [20], the annotator first watched the video and selected important actors (e.g., man or dog). For each actor they would then select a few key-frames that are representative of the actions performed by this actor throughout the video. They then described the actions of that entity with their voice, while at the same time moving their mouse over the key-frames to provide a spatio-temporal localization for each word. Finally, they transcribed their audio for a high-quality caption. For each actor, VidLN provides a long caption that often describes multiple actions. For the purpose of story visualization, we want to split the long caption into individual actions. Furthermore, we need timestamps for each action. The VidLN captions do not provide explicit timestamps for actions, but several words are localized in space-time on one or multiple key-frames. We use this indirect temporal information to derive an approximate timestamp.
Our pipeline consists of five steps (see Figure 2). 1) We prompt Flan-U-PaLM 540B [3, 56] – an instruction-tuned large language model (LLM) – to split a given description into multiple sentences, each (approximately) describing a single action, whilst making minimal (ideally, no) changes to the original words. 2) We can then map every word in each sentence to all (if any) of its associated
Figure 2: Automatic pipeline to transform VidLN annotations into stories. Starting from a video, its caption and annotated key-frames, we use an LLM to split the caption into multiple sentences. We then transfer the key-frames of the original caption into the new ones. We select a single key-frame per caption, merge captions with the same key-frame and finally split the video accordingly.
key-frames. As the LLM sometimes modifies words for better grammaticality, we rely on lemmas instead. 3) We associate a single key-frame to each caption, determined by the frame most commonly referenced by the verbs (if available), or across all words in the sentence. 4) If a continuous subsequence of sentences maps to the same key-frame, we concatenate those sentences into a single one. 5) Finally, we map each sentence to a time interval using the timestamp of its associated key-frame.
Our code to transform VidLN descriptions into stories is online. See App. D.5 for robustness statistics.
We train a compact (345M parameters) yet strong Phenaki [21] baseline for STORYBENCH, which we refer to as PHENAKI-GEN. It consists of a C-ViViT encoder, a text encoder, and a MaskGiT model.
The goal of the C-ViViT encoder is to learn a codebook of visual features to compress the videos in both temporal and spatial dimensions, while staying autoregressive in time. We train a 52M C-ViViT model on 27M Web-scraped video samples. Rather than a square resolution [21], we opt for a more natural, rectangular resolution of  $160 \times 96$  pixels. After being trained, the C-ViViT is kept frozen.
MaskGiT [57] is a bidirectional Transformer trained to predict multiple masked video tokens simultaneously conditioned on the caption features computed by the text encoder. Compared to the MaskGiT model used in the original Phenaki model (1.8B), ours is much smaller at 345M (24 layers). In contrast to the original Phenaki, which used a pretrained text encoder, we train the text encoder jointly with the video generation model on our collection of Web-collected 27M video captions. Our text encoder is a 12-layer Transformer that uses a T5 tokenizer [58]. Both models use 16 attention heads, MLP size of 3072, and default embedding size of 768. The maximum sequence length for the video component is 1440 tokens (11 compressed frames), and 64 tokens for the text component. The model is trained for 3M steps with a batch size of 256 using a fixed learning rate of 4.5e-5 after a linear warm-up for the first 10K steps using the Adam [59] optimizer. After pretraining, PHENAKI-GEN can generate arbitrarily long videos using the last 5 frames to condition the generation of the next 6.
We fine-tune PHENAKI-GEN on the training data of STORYBENCH for 500K steps with a much smaller batch size of 64 samples, using the same learning rate and optimizer of the pretraining stage.
Furthermore, we explore a novel variant of our model that is fine-tuned for continuation, PHENAKI-CONT, using the same hyperparameters as PHENAKI-GEN. Rather than training to predict all the input tokens, we only mask and generate the tokens of the last 6 frames, conditioning on 5 ground-truth frames. We hypothesize that this training would result in better transitions and consistency.
In this section, we discuss the performance of our baselines on STORYBENCH evaluated by humans and automatically. We append -ZS for results obtained in the zero-shot setting, -ST for single-task fine-tuning, and -MT for multi-task fine-tuning. In particular, we start by discussing performance on Oops-CSV (shown in Table 4) due to space limitations. We then also comment on the overall
PROMPT: The llama is putting his mouth in a white bowl in the car.
Figure 3: Comparison of PHENAKI-GEN-ZS to PHENAKI-CONT-ST on a prompt for action execution. While PHENAKI-GEN-ZS animates the animal, it adheres less to the text prompt, and the llama changes over time, PHENAKI-CONT-ST successfully shows two entities from the context (person and bowl), while keeping the animal and the surroundings consistent. Video subsampled by a factor 4.
PROMPTS:
A girl wearing a green-black bikini is walking on a diving board others are swimming
The girl slips on it and others get scared.
The girl falls in the sea while the other man is wearing glass and swimming
Figure 4: Applying PHENAKI-CONT-ST to a longer sequence with multiple prompts for story continuation. The model can generate the right action, including the water splash from the girl falling in the water. The background is kept relatively consistent within a short time frame, but it starts changing at longer time scales. The video was subsampled by a factor 4 to be shown here.
findings, reported in Figure 17 (App. E.1) for human evaluation, Tables 5 to 7 when using a set of neural metrics, and in Tables 8 to 10 (App. E.2) when using the others. Moreover, we also benchmark a variant of PHENAKI-CONT fine-tuned on the original Oops dataset, rather than after our automatic story-like transformation, which we refer to as PHENAKI-CONT-ST-orig. For every story, each model generates 4 output videos. We randomly sample one of them for human evaluation, but report mean and standard deviation across all 4 generated videos for automatic metrics. Due to our setup, all the results are at 8fps and using a  $160 \times 96$  pixel resolution for the generated videos, but note that ground-truth videos were processed at their original resolution for automatic evaluation. $^{4}$
6.1 Human Evaluation
Table 3 shows our human evaluation (c.f. Section 3.3) results for story continuation on Oops-CSV. Our model fine-tuned in continuation mode is vastly preferred over the zero-shot PHENAKI-GEN-ZS in all metrics. Figure 3 illustrates an example of why raters prefer PHENAKI-CONT-ST over PHENAKI-GEN-ZS. The improvement in long-term text adherence and consistency, however, has its limitations. We illustrate this in Figure 4: while the key actions are executed correctly, the background tends to drift over time, showing that our modest sized model has still large margins for improvement. We also see that human raters consistently prefer PHENAKI-CONT-ST over PHENAKI-GEN-ST (both fine-tuned on the same data), proving the effectiveness of the novel fine-tuning in continuation mode.
However, fine-tuning on all datasets jointly (PHENAKI-CONT-MT) results in videos of comparable quality as the zero-shot model. This is a common limitation of multi-task systems, and both a larger model and longer training could lead to better performance. Finally, we compare PHENAKI
For instance, the most common resolutions are  $1280 \times 720$  for Oops-CSV Dev set,  $854 \times 480$  for UVO-CSV Dev set, and  $360 \times 640$  for DiDeMo-CSV Dev set. We release our extracted features of the ground-truth videos for all the automatic evaluation metrics online at https://github.com/google/storybench.
Human Eval | Oops-CSV | Story Continuation L vs R	Visual quality	Text adherence	Entities consistency	BG consistency	Actions realism
L	Tie	R	L	Tie	R	L	Tie	R	L	Tie	R	L	Tie	R
PHENAKI-GEN-ZS vs PHENAKI-CONT-ST	37	14	49	34	8	58	29	15	56	16	19	65	29	13	58
PHENAKI-GEN-ST vs PHENAKI-CONT-ST	25	4	71	21	3	76	19	18	63	7	28	65	20	8	72
PHENAKI-GEN-ZS vs PHENAKI-CONT-MT	51	4	45	46	8	46	33	23	44	24	36	40	45	7	48
PHENAKI-GEN-ZS vs PHENAKI-CONT-STorig	53	4	43	41	10	49	33	28	39	29	35	36	38	7	55
Table 3: Human evaluation results for story continuation on Oops-CSV. For 100 randomly selected stories, we show the generated videos to 3 raters and report their majority vote. For each compared pair of models, L refers to the left model and R to the right model.
Oops-CSV
Model (@8 fps)	Action Execution	Story Continuation	Story Generation
FID↓	FVD↓	SIM↑	PQA↑	VTM↑	FID↓	FVD↓	SIM↑	PQA↑	VTM↑	FID↓	FVD↓	SIM↑	PQA↑	VTM↑
Zero-shot
PHENAKI-GEN-ZS	167	416	72.8	5.8	22.1	277	623	70.3	7.2	21.7	310	933	N/A	8.1	21.0
Single-Task (fine-tuned on Oops-CSV)
PHENAKI-GEN-ST	177	446	72.3	4.0	21.5	250	589	70.0	4.3	21.3	246	614	N/A	4.3	21.1
PHENAKI-CONT-ST	114	350	73.2	4.9	21.5	155	488	71.1	5.3	21.2	171	711	N/A	5.4	19.4
Multi-Task (fine-tuned jointly on Oops-CSV, DiDeMo-CSV, and UVO-CSV)
PHENAKI-CONT-MT	140	353	72.8	4.7	21.7	198	511	70.6	5.1	21.4	201	860	N/A	5.0	19.4
Table 4: Results from automatic evaluation metrics on Oops-CSV tasks. Best results are in bold. FID and SIM use InceptionV3, FVD uses I3D, PQA uses DOVER, and VTM uses CLIP.
CONT-ST-orig, fine-tuned on the original Oops VidLN annotations, against the zero-shot model. PHENAKI-CONT-ST-orig still outperforms the zero-shot model in all criteria except for visual quality. Compared to PHENAKI-CONT-ST, however, its margins over the zero-shot model are much smaller, clearly demonstrating the benefit of our training data transformation pipeline (c.f. Section 4).
In App. E.1, we report human ratings between PHENAKI-GEN-ZS and PHENAKI-CONT-ST on action recognition and story generation. For action recognition, we find consistent results with story continuation. In contrast, the zero-shot model is generally preferred for story generation. We hypothesize that fine-tuning for continuation leads to a less capable model for the task of generation from scratch. Unifying both approaches in a single model is an interesting direction for future work.
6.2 Automatic Evaluation
While the results from human raters were very clear, they are expensive to obtain. We hence also report typical automatic metrics on Oops-CSV in Table 4, and on all datasets in Tables 5 to 7.
In Table 4, our model fine-tuned on Oops-CSV in continuation mode (PHENAKI-CONT-ST) performs the best on action execution and story continuation according to FID, FVD, and SIM, again demonstrating the effectiveness of the novel continuation mode. On the other hand, for story generation, the continuation mode yields an improvement in FID but not in FVD. This contrasts with the human raters, who preferred PHENAKI-GEN-ZS in all metrics for story generation (see Figure 17 in App. E.1).
Across all datasets and tasks, PHENAKI-CONT-MT achieves overall strong results, showcasing the effectiveness of training a single text-to-video model in continuation mode on all datasets. On the other hand, PHENAKI-GEN-ZS achieves higher PQA scores, while human raters typically found all the models to have similar visual quality. This, however, is partially reflected in our SIM score, which only varies by a small amount between different models (all within 1pp). Nevertheless, while human raters vastly preferred PHENAKI-CONT-ST over PHENAKI-GEN-ZS, the SIM, PQA and VTM metrics do not reflect this, showing low correlation of automatic metrics with human judgments.
STORYBENCH is the first effort towards a multifaceted benchmark for text-to-video generation. It aims at fostering progress on real-world video generation, with dynamic text prompts and controllable duration provided by the users. Yet, it has a few notable limitations. First, while our benchmark tests for multi-sentence generation, the annotated videos are not very long. Second, they are mostly single-shot (i.e., without changes of scene) and user-generated (rather than professional), which
<sup>5</sup>We find similar results with newer neural metrics for automatic evaluation in Tables 8 to 10 in App. E.2.
Action Execution Model (@8 fps)	Oops-CSV	UVO-CSV	DiDeMo-CSV
FIDiv3↓	FVDI3D↓	SIMiv3↑	PQA↑	VTMC↑	FIDiv3↓	FVDI3D↓	SIMiv3↑	PQA↑	VTMC↑	FIDiv3↓	FVDI3D↓	SIMiv3↑	PQA↑	VTMC↑
Zero-Shot
PHENAKI-GEN-ZS	167±3.53	416±8.78	72.8±0.04	5.8±0.03	22.1±0.05	110±0.97	472±8.03	73.0±0.02	8.5±0.1	22.4±0.03	195±1.57	454±3.21	70.7±0.03	6.7±0.02	22.1±0.02
Single-Task
PHENAKI-GEN-ST	177±0.60	446±3.47	72.3±0.01	4.0±0.02	21.5±0.00	165±0.68	1004±8.55	72.0±0.01	4.8±0.05	20.1±0.02	159±1.73	510±7.09	70.8±0.02	4.5±0.02	22.3±0.01
PHENAKI-CONT-ST	114±1.10	350±2.93	73.2±0.01	4.8±0.02	21.5±0.02	178±1.74	828±5.98	72.0±0.01	5.6±0.02	20.5±0.04	136±0.80	365±2.56	71.2±0.01	5.6±0.01	22.1±0.02
Multi-Task
PHENAKI-GEN-MT	270±3.11	502±4.65	71.5±0.02	3.8±0.04	21.4±0.03	235±2.09	916±4.47	71.4±0.02	5.1±0.05	20.6±0.02	219±1.65	510±1.86	70.2±0.02	4.7±0.06	21.9±0.02
PHENAKI-CONT-MT	140±1.21	353±2.83	72.8±0.02	4.7±0.01	21.7±0.03	111±2.01	435±4.89	72.8±0.03	6.8±0.01	21.6±0.03	166±3.00	381±2.96	70.9±0.02	5.8±0.03	21.9±0.02
Table 5: Results from automatic evaluation metrics on action execution tasks. Best results are in bold. FID and SIM use InceptionV3, FVD uses I3D, PQA uses DOVER, and VTM uses CLIP.
Story Continuation
Model (@8 fps)	Oops-CSV	UVO-CSV	DiDeMo-CSV
FIDiv3↓	FVD13D↓	SIMiv3↑	PQA↑	VTMC↑	FIDiv3↓	FVD13D↓	SIMiv3↑	PQA↑	VTMC↑	FIDiv3↓	FVD13D↓	SIMiv3↑	PQA↑	VTMC↑
Zero-Shot
PHENAKI-GEN-ZS	277±4.04	623±9.93	70.3±0.02	7.2±0.06	21.7±0.02	138±1.79	660±8.20	72.0±0.02	9.4±0.09	22.2±0.04	341±1.89	817±7.76	68.0±0.01	7.3±0.07	21.6±0.02
Single-Task
PHENAKI-GEN-ST	250±2.05	589±6.08	70.0±0.02	4.3±0.02	21.3±0.01	193±1.20	1267±9.36	71.0±0.01	5.0±0.03	19.9±0.01	238±1.24	690±1.90	68.2±0.02	4.0±0.02	22.0±0.01
PHENAKI-CONT-ST	155±1.07	488±2.12	71.1±0.03	5.3±0.04	21.2±0.04	197±1.22	1125±9.84	71.1±0.02	5.7±0.03	20.2±0.03	206±1.95	580±4.30	68.7±0.01	5.4±0.03	21.6±0.04
Multi-Task
PHENAKI-GEN-MT	364±2.37	708±7.29	69.0±0.01	4.2±0.02	21.1±0.02	268±2.76	1188±8.76	70.3±0.02	5.2±0.03	20.5±0.02	330±2.09	717±9.12	67.5±0.01	4.3±0.04	21.5±0.02
PHENAKI-CONT-MT	198±1.97	511±5.46	70.6±0.04	5.1±0.02	21.4±0.02	124±2.56	618±6.59	71.9±0.03	7.0±0.02	21.3±0.03	256±3.27	684±3.15	68.3±0.03	5.5±0.09	21.2±0.01
Table 6: Results from automatic evaluation metrics on story continuation tasks. Best results are in bold. FID and SIM use InceptionV3, FVD uses I3D, PQA uses DOVER, and VTM uses CLIP.
Story Generation
Model (@8 fps)	Oops-CSV	UVO-CSV	DiDeMo-CSV
FIDiv3↓	FVD13D↓	SIMiv3↑	VTMC↑	FIDiv3↓	FVD13D↓	SIMiv3↑	PQA↑	VTMC↑	FIDiv3↓	FVD13D↓	SIMiv3↑
Zero-Shot
PHENAKI-GEN-ZS	310±3.13	933±4.50	N/A	8.1±0.03	21.0±0.05	181±3.87	1118±6.13	N/A	10.0±0.06	20.8±0.09	357±2.79	930±8.71
Single-Task
PHENAKI-GEN-ST	246±1.07	614±6.86	N/A	4.2±0.02	21.1±0.03	188±1.45	1371±6.06	N/A	4.9±0.01	19.8±0.01	241±0.90	710±3.54
PHENAKI-CONT-ST	171±5.32	711±9.55	N/A	5.4±0.03	19.4±0.02	207±2.75	1600±9.97	N/A	5.4±0.02	19.1±0.04	232±3.11	689±5.69
Multi-Task
PHENAKI-GEN-MT	356±4.36	760±4.60	N/A	4.1±0.02	21.1±0.04	268±2.50	1311±7.41	N/A	5.1±0.03	20.3±0.02	334±1.16	739±4.49
PHENAKI-CONT-MT	201±5.56	860±9.86	N/A	5.0±0.11	19.4±0.05	148±5.50	1150±9.98	N/A	6.3±0.08	18.8±0.06	269±2.80	863±9.94
Table 7: Results from automatic evaluation metrics on story generation tasks. Best results are in bold. FID and SIM use InceptionV3, FVD uses I3D, PQA uses DOVER, and VTM uses CLIP.
differ from realistic filmmaking applications. Moreover, we highlight that STORYBENCH focuses on real-world videos, and does not include artistic content never seen in the real world. Yet, we expect it can drive general progress in text-to-video generation thanks to its complexity. Our annotation framework would not scale to long videos, and we encourage future work to investigate more efficient protocols. We also note that our baseline models are small (345M) compared to current text-to-image models (20B). Effectively training large text-to-video models is an open question. Finally, we also found that automatic metrics do not fully align with human ratings. This might indicate that the metrics are suboptimal, but we cannot exclude that they would perform better on higher resolution videos. This highlights the need for a future in-depth study on human versus automated evaluation.
In this work, we collected annotations for the task of video generation from a sequence of text prompts narrating the video evolution (i.e., continuous story visualization). Our data is very rich, with timestamps for each text prompt, as well as diagnostic labels for each video segment. On top of such unique data, we propose STORY BENCH: a new benchmark to measure progress of forthcoming text-to-video models on different tasks, three datasets and in three evaluation setups. STORY BENCH allows to evaluate key challenges of arbitrarily long video generation, such as consistency over time and realistic action synthesis. We benchmarked small yet strong baselines, and showed that fine-tuning a Phenaki model for continuation improves such aspects. Our results highlight a discrepancy between human and automatic ratings. While human evaluation remains the most reliable method, we encourage future work to also report automatic metrics in an effort to better understand and develop effective automatic evaluation. Our benchmark enables a systematic comparison of text-to-video models that can perform story generation with prompts that vary over time and specified duration. Beyond modeling, opportunities for future work include expanding the benchmark with datasets of longer videos and richer variety (e.g., camera jumps, scene changes, etc.) to enhance its applicability to industrial-level videos, and devising efficient data annotations protocols to facilitate the collection.
We would like to thank Irina Blok, Alonso Martinez, Mario Lucic, Cordelia Schmid, Jasper Uijlings, Andreas Steiner and Lucas Beyer for helpful discussions, as well as the annotators for their work.
[56] Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. ArXiv, abs/2210.11416, 2022.
[57] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11315-11325, June 2022.
[58] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[59] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[60] Rowan T. Hughes, Liming Zhu, and Tomasz Bednarz. Generative adversarial networks-enabled human-artificial intelligence collaborative applications for creative and design industries: A systematic review of current approaches and trends. Frontiers in Artificial Intelligence, 4, 2021.
[61] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy, July 2019. Association for Computational Linguistics.
[62] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021.
[63] Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1536-1546, 2021.
[64] Nicole Meister, Dora Zhao, Angelina Wang, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. Gender artifacts in visual datasets. arXiv preprint arXiv:2206.09191, 2022.
[65] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[66] Xiyang Luo, Ruohan Zhan, Huiwen Chang, Feng Yang, and Peyman Milanfar. Distortion agnostic deep watermarking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
We bear all responsibilities for the content, licensing, distribution, and maintenance of our datasets in STORYBENCH. Our datasets are released under a CC-BY-4.0 license, and our code under an Apache license. Data, code and annotation guidelines are hosted on GitHub at the following URL: https://github.com/google/storybench.
The aim of STORYBENCH is to enable reliable measurements of progress in generative text-to-video models. While this kind of models have great potential to assist and augment human creativity [60], there are broader societal issues that need to be considered when developing these models.
First, while we annotate an evaluation set, training current, strong text-to-video models is computationally expensive. This affects not only their financial cost (e.g., hardware and electricity), but also their environmental cost due to the carbon footprint of modern tensor processing hardware [61].
Second, massive amounts of data are required to train state-of-the-art generative models. Such datasets are harvested from the Web, which tend to reflect social stereotypes, oppressive viewpoints, and harmful associations to marginalized identity groups [62-64]. Other biases include those introduced by the use of examples that primarily have English texts and may reflect North American and Western European cultures [65]. We expect models trained on them to reflect these biases, and hence caution developers to assess the limitations of their models before integrating them into user-facing applications. To facilitate positive and safe integration of text-to-video models, we encourage future work to create benchmark evaluations to assess social and cultural biases of these technologies.