# Text-to-video model

A text-to-video model is a machine learning model that uses a natural language description as input to produce a video relevant to the input text. Advancements during the 2020s in the generation of high-quality, text-conditioned videos have largely been driven by the development of video diffusion models.

## Models

There are different models, including open source models. Chinese-language input CogVideo is the earliest text-to-video model "of 9.4 billion parameters" to be developed, with its demo version of open source codes first presented on GitHub in 2022. That year, Meta Platforms released a partial text-to-video model called "Make-A-Video", and Google's Brain (later Google DeepMind) introduced Imagen Video, a text-to-video model with 3D U-Net.
In March 2023, a research paper titled "VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation" was published, presenting a novel approach to video generation. The VideoFusion model decomposes the diffusion process into two components: base noise and residual noise, which are shared across frames to ensure temporal coherence. By utilizing a pre-trained image diffusion model as a base generator, the model efficiently generated high-quality and coherent videos. Fine-tuning the pre-trained model on video data addressed the domain gap between image and video data, enhancing the model's ability to produce realistic and consistent video sequences. In the same month, Adobe introduced Firefly AI as part of its features.
In January 2024, Google announced development of a text-to-video model named Lumiere which is anticipated to integrate advanced video editing capabilities. Matthias Niessner and Lourdes Agapito at AI company Synthesia work on developing 3D neural rendering techniques that can synthesise realistic video by using 2D and 3D neural representations of shape, appearances, and motion for controllable video synthesis of avatars. In June 2024, Luma Labs launched its Dream Machine video tool. That same month, Kuaishou extended its Kling AI text-to-video model to international users. In July 2024, TikTok owner ByteDance released Jimeng AI in China, through its subsidiary, Faceu Technology. By September 2024, the Chinese AI company MiniMax debuted its video-01 model, joining other established AI model companies like Zhipu AI, Baichuan, and Moonshot AI, which contribute to China’s involvement in AI technology.
Alternative approaches to text-to-video models include Google's Phenaki, Hour One, Colossyan, Runway's Gen-3 Alpha, and OpenAI's Sora, Several additional text-to-video models, such as Plug-and-Play, Text2LIVE, and TuneAVideo, have emerged. Google is also preparing to launch a video generation tool named Veo for YouTube Shorts in 2025. FLUX.1 developer Black Forest Labs has announced its text-to-video model SOTA.

## Architecture and training

There are several architectures that have been used to create Text-to-Video models. Similar to Text-to-Image models, these models can be trained using Recurrent Neural Networks (RNNs) such as long short-term memory (LSTM) networks, which has been used for Pixel Transformation Models and Stochastic Video Generation Models, which aid in consistency and realism respectively. An alternative for these include transformer models. Generative adversarial networks (GANs), Variational autoencoders (VAEs), — which can aid in the prediction of human motion — and diffusion models have also been used to develop the image generation aspects of the model.
Text-video datasets used to train models include, but are not limited to, WebVid-10M, HDVILA-100M, CCV, ActivityNet, and Panda-70M. These datasets contain millions of original videos of interest, generated videos, captioned-videos, and textual information that help train models for accuracy. Text-video datasets used to train models include, but are not limited to PromptSource, DiffusionDB, and VidProM. These datasets provide the range of text inputs needed to teach models how to interpret a variety of textual prompts.
The video generation process involves synchronizing the text inputs with video frames, ensuring alignment and consistency throughout the sequence. This predictive process is subject to decline in quality as the length of the video increases due to resource limitations.

## Limitations

Despite the rapid evolution of Text-to-Video models in their performance, a primary limitation is that they are very computationally heavy which limits its capacity to provide high quality and lengthy outputs. Additionally, these models require a large amount of specific training data to be able to generate high quality and coherent outputs, which brings about the issue of accessibility.
Moreover, models may misinterpret textual prompts, resulting in video outputs that deviate from the intended meaning. This can occur due to limitations in capturing semantic context embedded in text, which affects the model’s ability to align generated video with the user’s intended message. Various models, including Make-A-Video, Imagen Video, Phenaki, CogVideo, GODIVA, and NUWA, are currently being tested and refined to enhance their alignment capabilities and overall performance in text-to-video generation.

## Ethics

The deployment of Text-to-Video models raises ethical considerations related to content generation. These models have the potential to create inappropriate or unauthorized content, including explicit material, graphic violence, misinformation, and likenesses of real individuals without consent. Ensuring that AI-generated content complies with established standards for safe and ethical usage is essential, as content generated by these models may not always be easily identified as harmful or misleading. The ability of AI to recognize and filter out NSFW or copyrighted content remains an ongoing challenge, with implications for both creators and audiences.

## Impacts and applications

Text-to-Video models offer a broad range of applications that may benefit various fields, from educational and promotional to creative industries. These models can streamline content creation for training videos, movie previews, gaming assets, and visualizations, making it easier to generate high-quality, dynamic content. These features provide users with economical and personal benefits.
The feature film The Reality of Time, the world's first full-length movie to fully integrate generative AI for video, was completed in 2024. Narrated in part by John de Lancie (known for his iconic role as "Q" in Star Trek: The Next Generation). Its production utilized advanced AI tools, including Runway Gen-3 Alpha and Kling 1.6, as described in the book Cinematic A.I. The book explores the limitations of text-to-video technology, the challenges of implementing it, and how image-to-video techniques were employed for many of the film's key shots.

## Comparison of existing models

| Model/Product | Company | Year released | Status | Key features | Capabilities | Pricing | Video length | Supported languages |
| Synthesia | Synthesia | 2019 | Released | AI avatars, multilingual support for 60+ languages, customization options | Specialized in realistic AI avatars for corporate training and marketing | Subscription-based, starting around $30/month | Varies based on subscription | 60+ |
| InVideo AI | InVideo | 2021 | Released | AI-powered video creation, large stock library, AI talking avatars | Tailored for social media content with platform-specific templates | Free plan available, Paid plans starting at $16/month | Varies depending on content type | Multiple (not specified) |
| Fliki | Fliki AI | 2022 | Released | Text-to-video with AI avatars and voices, extensive language and voice support | Supports 65+ AI avatars and 2,000+ voices in 70 languages | Free plan available, Paid plans starting at $30/month | Varies based on subscription | 70+ |
| Runway Gen-2 | Runway AI | 2023 | Released | Multimodal video generation from text, images, or videos | High-quality visuals, various modes like stylization and storyboard | Free trial, Paid plans (details not specified) | Up to 16 seconds | Multiple (not specified) |
| Pika Labs | Pika Labs | 2024 | Beta | Dynamic video generation, camera and motion customization | User-friendly, focused on natural dynamic generation | Currently free during beta | Flexible, supports longer videos with frame continuation | Multiple (not specified) |
| Runway Gen-3 Alpha | Runway AI | 2024 | Alpha | Enhanced visual fidelity, photorealistic humans, fine-grained temporal control | Ultra-realistic video generation with precise key-framing and industry-level customization | Free trial available, custom pricing for enterprises | Up to 10 seconds per clip, extendable | Multiple (not specified) |
| OpenAI Sora | OpenAI | 2024 | Alpha | Deep language understanding, high-quality cinematic visuals, multi-shot videos | Capable of creating detailed, dynamic, and emotionally expressive videos; still under development with safety measures | Pricing not yet disclosed | Expected to generate longer videos; duration specifics TBD | Multiple (not specified) |