翻译下面的内容，并给出详细解释或代码
3.4.2 Improving Inference Efficiency To sample videos efficiently, we use the Euler sampler with a unique t-schedule tailored to our model. Empirically, we found that Euler outperforms higher-order solvers like midpoint (Atkinson, 1991) or adaptive solvers like Dopri5 (Dormand and Prince, 1980). We observed that reducing the number of inference steps for video generation is more challenging than for image generation due to the additional time dimension, i.e., the quality and prompt alignment of the generated motion are more sensitive to the number of inference steps compared to static images. For example, videos generated with 250, 500, or 1000 linear steps show noticeable differences in scene composition and motion quality. While techniques such as distillation (Salimans and Ho, 2022; Kohler et al., 2024) can be used to speed up the model inference, they require additional training. Next, we show a simple inference-only technique that can lead up to ⇠ 20⇥ speed up with a few lines of code. We found that we can closely approximate the quality of an N-step video generation process with merely 50 steps by implementing a linear-quadratic t-schedule. This approach follows the first 25 steps of an N-step linear schedule and then approximates the remaining N 25 steps with 25 quadratically placed steps. For example, a video generated with 1000 linear steps can be precisely emulated by 25 linear steps followed by 25 quadratic steps, whereby the linear steps are identical to the first 25 linear steps of a 1000-step linear schedule. The linear-quadratic strategy is predicated on the observation that the first inference steps are pivotal in setting up the scene and motion of the video. This is visualized in Figure 10, where we plot the average change between the input and output of each transformer block at every inference step. Similar behavior is observed in the diffusion-based fast video model PAB (Zhao et al., 2024), where the average per-step difference of attention blocks follows a U-shaped pattern, compared to the L-shaped curve in Figure 10. Since most changes occur in the first solver steps, taking over the first linear steps of an N-step schedule followed by much bigger steps is enough to approximate the full N-step result. The quadratic spacing of the latter steps is crucial, as it emphasizes the importance of the early stages in the flow-matching sequence. In practice, we use a 50-step linear-quadratic schedule emulating N = 250 linear steps for optimal results.
3.5 Evaluation In this section, we explain how we evaluate the text-to-video quality of Movie Gen Video and other models. Our goal is to establish clear and effective evaluation metrics that identify a model’s weaknesses and provide reliable feedback. We explain the different text-to-video evaluation axes and their design motivations in Section 3.5.1. We introduce our new benchmark, Movie Gen Video Bench, in Section 3.5.2. Throughout this work, we use human evaluation to assess the quality of generated videos across various evaluation axes. When evaluating each axis, we conduct pairwise A/B tests where expert human evaluators assess two videos side-by-side. Evaluators are instructed to choose a winner based on the axis being measured, or to declare a tie in case of no clear winner. We include a discussion on the motivation and reliability of using human evaluations and on existing automated metrics in Section 3.5.3. 3.5.1 Evaluation Axes Evaluating text-to-video generation presents unique challenges compared to text-to-image tasks, primarily due to the added complexity of the temporal dimension. For a video to be considered high quality, it must stay faithful to the provided text prompt, maintain a high visual quality across frames without noticeable flaws, and be visually appealing with a photorealistic style. To assess these factors, we evaluate the quality of generated videos across three main axes: (1) Text-alignment, (2) Visual quality, and (3) realness and aesthetics. Each axis, along with their fine-grained sub-axes, is described in details below and summarized in Table 4. Text-alignment. This axis measures how well a generated video aligns with the provided prompt. An input prompt can include wide-ranging descriptions of subject appearance, motion, background, camera motion, lighting and style, visual text, etc. Human evaluators are asked to pay close attention to these specific aspects and select the video that aligns more closely with the prompt. To provide more nuanced feedback, evaluators Text-alignment Visual quality Realness & Aesthetics Subject & Motion alignment Overall Frame consistency Motion Completeness Motion Naturalness Realness Aesthetics Table 4 Evaluation axes for text-to-video generation. We evaluate video generations across 3 axes, each of which is composed of multiple fine-grained sub-axes. Text-alignment evaluates the ‘alignment’ between the input text prompt and the video. Visual quality, Realness & Aesthetics evaluate the quality of the generated video independent of the input text prompt.
are also asked to specify their reasoning based on two orthogonal sub-axes: Subject match: This measures alignment of subject appearance, background, lighting and style; and Motion match: This measures the alignment of motion-related descriptions. Visual Quality. Compared to visual quality in text-to-image generation, much of the perceived quality in generated videos stems from the quality of motion – a video-specific dimension. Therefore, in text-to-video visual quality evaluation, we focus on measuring the model’s ability to generate consistent, natural, and sufficient amounts of motion in the output videos. To capture these critical aspects, we propose the following four sub-axes, which we outline below. • Frame consistency: This metric assesses the temporal consistency of generated content. Violations of frame consistency can manifest as morphing-like artifacts, blurred or distorted objects, or content that abruptly appears or disappears. We consider frame consistency a crucial measure of the model’s ability to understand object framing and relationships in motion, as inconsistencies or distortions often arise when the model fails to accurately represent interactions between objects or their environment. Additionally, frame consistency reflects the model’s capacity to handle challenging tasks, such as prompts that require fast-moving content, e.g., in sports scenarios, where maintaining consistent appearance is especially difficult; or reasoning about occlusions, e.g., objects re-appearing after being occluded. • Motion completeness: This measures whether the output video contains enough motion. A lack of motion completeness may occur when the prompt involves out-of-distribution or unusual subjects (e.g., monsters, ghosts) or real-world objects performing unusual activities (e.g., people flying, pandas playing piano). Due to limited training data for such scenarios, the model may struggle to generate enough amount of motion, resulting in either static videos or those with only camera movement. Motion completeness evaluates the magnitude of motion in the video. A win on this axis indicates a greater amount of motion, even if it includes distortion, fast motion, or appears unnatural. • Motion naturalness: This metric assesses the model’s ability to generate natural and realistic motion, demonstrating a solid understanding of real-world physics. It covers aspects such as natural limb movements, facial expressions, and adherence to physical laws. Motion that appears unnatural or uncanny will be penalized. • Overall quality: For a given pair of videos being compared, the above three metrics might not result in the same winner. To resolve this, we introduced the overall quality sub-axis, where human evaluators are asked to pick the winning video that has better “overall” quality given the previous three sub-axes. This is a holistic metric that asks the human annotators to use their perception and to balance the previous signals to capture overall how good a generated video is. Realness & Aesthetics. Realness and aesthetics evaluate the model’s ability to generate photorealistic videos with aesthetically pleasing content, lighting, color, style, etc. We ask human evaluators to evaluate along two sub-axes: • Realness: This measures which of the videos being compared most closely resembles a real video. For fantastical prompts that are out of the training set distribution (e.g., depicting fantasy creatures or surreal scenes), we define realness as mimicking a clip from a movie following a realistic art-style. We additionally ask the evaluators to select a reason behind their choice i.e., “subject appearance being more realistic” or “motion being more realistic”. • Aesthetics: This measures which of the generated videos has more interesting and compelling content, lighting, color, and camera effects. Again, we ask the evaluators to provide details justifying their choice from “content being more appealing/interesting”, and “lighting/color/style being more pleasing”.
3.5.2 Evaluation benchmark In order to thoroughly evaluate video generations, we propose and hope to release a benchmark, Movie Gen Video Bench, which consists of 1000 prompts that cover all the different testing aspects summarized above. Our benchmark is more than 3⇥ larger than the prompt sets used in prior work (Singer et al., 2023; Girdhar et al., 2024) for evaluation. We specifically include prompts capturing the following concepts of interest: 1) human activity (limb and mouth motion, emotions, etc.), 2) animals, 3) nature and scenery, 4) physics (fluid Prompt Concept Motion level A marathon runner crossing the finish line after a grueling race. human activity high A curious cat peering out from a cozy hiding spot. animals medium A peaceful countryside with rolling hills and a setting sun. nature & scenery low A high-speed video of champagne being poured into a glass, with bubbles rising rapidly. physics – fluid dynamics high A cute golden dragon walking like a model on stage, as the audience claps for him. unusual subjects low Table 5 Sample evaluation prompts from our proposed Movie Gen Video Bench benchmark. We sample prompts that cover a wide range of concepts, with varying motion levels.Figure 11 Summary visualization of evaluation prompts in Movie Gen Video Bench. (a) Since each prompt may encompass multiple testing concepts, we assign one or more concept tags to each prompt. As a result, the total count in the distribution exceeds 1000. (b), (c) Visualization of the common nouns and verbs in the evaluation set, respectively. dynamics, gravity, acceleration, collisions, explosions, etc.), 5) unusual subjects and unusual activities. To test the generation quality at different motion levels, we also tag each prompt with high/medium/low motion. We show examples of evaluation prompts used in Table 5 and show the distribution of evaluation prompts across concepts in Figure 11. We evaluate the model quality on the entire evaluation prompt set as well as break down the quality by individual testing metrics. Prompts involving unusual subjects and unusual motion help test model’s ability to generalize to out-of-distribution content. 3.5.3 Evaluation Discussion Here, we motivate our decision for using human evaluation as opposed to automated metrics. Necessity of human evaluations for video generation. The motivation for choosing human evaluation stems from the complexity of evaluating video generation. For Text-alignment, evaluating the alignment of motion over time requires understanding how actions unfold and evolve in relation to the prompt. Humans are particularly skilled at recognizing temporal coherence and handling ambiguities when the context is abstract or complex, whereas automated methods may only capture static frame-level correspondences. When evaluating visual quality, such as motion naturalness or detecting inconsistencies in object appearance across frames, humans excel due to their innate understanding of real-world physics and object behavior. Similarly, assessing realness and aesthetics heavily depends on human perception and preference. Across all three tasks, we find that existing automated metrics struggle to provide reliable results, reinforcing the need for human evaluation.
Reliability. An important aspect concerning reliability in evaluations is the randomness introduced both on the modeling side due to the probabilistic nature of the generative models, and the human evaluation side due to the annotation variance. Defining objective criteria to measure generations remains challenging and humans can still be influenced by other factors such as personal biases or preferences. We describe our efforts to reduce evaluation variance and increase the reliability of the human evaluations. We take four key steps towards minimizing evaluation variance: (1) We provide human evaluators with detailed evaluation guidelines and video examples, narrowing the definitions of evaluation axes and sub-axes to minimize subjectivity. Additionally, inspired by the JUICE metric (Girdhar et al., 2024), we found that asking evaluators to indicate the reason of their choices helps reduce annotation variance and improve agreement among evaluators. (2) We evaluate the models over a large set of prompts (e.g., 1000 for Movie Gen Video Bench, 3⇥ larger than (Singer et al., 2023; Girdhar et al., 2024)) covering a wide variety of concepts. (3) We use a majority vote system, with a majority vote from three annotations for each Text-alignment and Visual quality question, and a majority vote from six annotations for realness and aesthetic questions, as these are more subjective. (4) We conduct thorough and frequent audits of human annotations to resolve edge cases and correct mislabelings. Automated Metrics for text-to-video evaluation. Prior works in text-to-video generation have relied upon automated metrics for assessing video quality. Similar to recent studies (Dai et al., 2023; Podell et al., 2023; Singer et al., 2023; Girdhar et al., 2024; Ho et al., 2022a; Barratt and Sharma, 2018; Chong and Forsyth, 2020; Ge et al., 2024; Huang et al., 2024) we find that automated metrics such as FVD (Unterthiner et al., 2019) and IS (Salimans et al., 2016) do not correlate with human evaluation scores for video quality, and hence do not provide useful signal for model development or comparison. Some prior works utilize discriminative models for generated media evaluation axes, including text faithfulness with CLIP (Radford et al., 2021). One key limitation of such automated metrics is that they are inherently limited by the performance of the underlying discriminative model (Rambhatla and Misra, 2023). A key challenge in using discriminative models for evaluating text-to-video generation is the inavailability of sufficiently effective and expressive video-text discriminative models. We note that other interesting automated metrics for generated video evaluation exist, such as those based on structure-from-motion (Li et al., 2024), that we did not explore the use of here. Enabling fair comparison to Movie Gen Video. To enable fair and easy comparison to Movie Gen Video for future works, we hope to publicly release our non cherry picked generated videos for the Movie Gen Video Bench prompt set.
3.6 Results In this section, we describe the experiments and results for Movie Gen Video. We first include comparisons to prior work for text-to-video generation in Section 3.6.1. We ablate key design decisions for Movie Gen Video in Section 3.6.2. We include key results and ablations for the TAE in Section 3.6.3, and an evaluation of the Spatial Upsampler in Section 3.6.5. We include comparisons to prior work for text-to-image generation in Section 3.7. 3.6.1 Comparisons to Prior Work Where possible, we obtain non-cherry picked generated videos from the Movie Gen Video Bench prompt set for the prior work methods, and compare to these using non-cherry picked videos from Movie Gen Video for the same prompts. This includes the black-box commercial models that offer API access through their website: Runway Gen3 (RunwayML, 2024), LumaLabs (LumaLabs, 2024), Kling1.5 (KlingAI, 2024). We also compare to closed source text-to-video methods (OpenAI Sora), where our only option is to compare to them using the prompts and videos from their publicly released examples. Note that the publicly released videos for closed source methods are likely to be ‘best’ representative samples obtained through cherry picking. Hence for fair comparison, we compare to OpenAI Sora by methodically manually choosing one video from five generated options from Movie Gen Video for each prompt. One additional challenge when comparing to prior work is that each method generates videos at different resolutions and aspect-ratios. We reduce annotator bias (Girdhar et al., 2024) by downsampling Movie Gen Video’s videos for each comparison such that they match in these aspects. Full details on this postprocessing and the OpenAI Sora comparison can be found in Appendix C.2.
We compare Movie Gen Video to prior work for text-to-video generation on different evaluation axes 20
Table 6 Movie Gen Video vs. prior work. The comparison uses either generated videos from the Movie Gen Video Bench prompt set (Runway Gen3, LumaLabs, Kling1.5) or prompts from publicly released videos on their website (OpenAI Sora). A detailed summary of information from prior work is shown in Table 40. We measure the net win rate (win% - loss% of our model) which has a range of [100%, 100%]. To assess statistical significance, we perform an annotation variance analysis (Appendix C.1), with the net win standard deviation, , indicated in the table above. A significant win/loss is identified when the net win rate is beyond 2 (95% CI), a moderate win/loss within 1–2 (68% CI), and performance is considered on par within 1. described in Section 3.5. The results are shown in Table 6. We report the net win rate of our model, which can lie in the range [100, 100]. On overall quality, Movie Gen Video strongly outperforms Runway Gen3 (35.02%) and LumaLabs with the net win rate beyond 2. Our generations moderately net win over OpenAI Sora (8.23%) (net win rate within 1-2 ) and are on par with Kling1.5 (3.87%). Against Runway Gen3, LumaLabs and OpenAI Sora, we see that Movie Gen Video either outperforms or is on par across all quality breakdown axes, including large net wins against Runway Gen3 on motion naturalness (19.27%) and frame consistency (33.1%), against Sora on frame consistency (8.22%) and motion completeness (8.86%). These significant net wins demonstrate Movie Gen Video’s ability to simulate the real world with generated videos that respect physics, with motion that is both reasonable in magnitude but consistent and without distortion. Against Kling1.5, we see that Movie Gen Video significantly net wins in frame consistency (13.5%) but loses on motion completeness (-10.04%). We note that this large motion completeness paired with poor frame consistency shows Kling1.5’s tendency to occasionally generate unnaturally large motion with distortion. As indicated in Section 3.5.1, motion completeness only evaluates the magnitude of motion in the video, regardless of distortion, fast motion, or being unnatural. On realness and aesthetics, Movie Gen Video significantly outperforms Runway Gen3, LumaLabs and Kling1.5 on both metrics, with 48.49%, 61.83% and 37.09% net win rates on realness, respectively. Compared to OpenAI Sora, Movie Gen Video has a significant win on realness with 11.62% net win rate beyond 2 and a moderate win over OpenAI Sora on aesthetics with 6.45% net win rate within 1–2 . This demonstrates Movie Gen Video’s ability to generate photorealistic and visually compelling content. For text faithfulness, Movie Gen Video outperforms OpenAI Sora, Runway Gen3, LumaLabs and is on par with Kling1.5. Several generated videos from Movie Gen Video are shown in Figure 12. Movie Gen Video is able to generate high quality videos for both natural prompts (see Figure 12) and out-of-distribution prompts describing fantastical scenes from outside of the training set distribution (see Figure 1). The generated videos contain complex motion, depicting detailed content over the video’s duration e.g., a firefighter running into and then out of a burning forest or a puppy searching for, finding its owner, and continuing its quest (see Figure 12). Qualitative comparisons between Movie Gen Video and prior work are shown in Figure 13 and Figure 14. As shown, Movie Gen Video generates realistic and high quality videos with natural-looking motion that is well aligned to the text prompt. Movie Gen Video generates objects and identities that are consistent over the entire duration of the video, and that obey the laws of physics. Differently, the prior work can struggle to generate videos that are simultaneously high quality and with good text-alignemnt.

Figure 14 Qualitative comparisons between Movie Gen Video and prior work. Here, we show two generated videos for the same prompt for Movie Gen Video and OpenAI Sora. Movie Gen Video generates natural-looking videos with realistic motion, even for the out-of-training-set-distribution prompts shown here. As shown here, for such prompts OpenAI Sora can tend to generate less realistic videos (e.g., the cartoonish kangaroo in the second row), that can be missing the motion details described in the text prompt (e.g., the non-walking robot in the bottom row). Videos in this Figure found at https://go.fb.me/MovieGen-Figure14. Correlation between validation loss and human evaluation. In Figure 15, we show the validation loss for Movie Gen Video as a function of pretraining steps and observe that it decreases smoothly. We take pre-trained checkpoints after every few thousand iterations and evaluate them in a pairwise comparison. We observe that the validation loss is well correlated with human evaluation results as the later checkpoints with lower validation loss perform better in the evaluations. This suggests that the Flow Matching validation loss can serve as a useful proxy for human evaluations during model development. Effect of finetuning. We leverage supervised finetuning, described in Section 3.3 to further improve the video generation quality. In Table 7, we compare the evaluation metrics between pre-trained model and finetuned models at 24 FPS with 10.6s video duration. We find that finetuning leads to a significant improvement on both the Visual quality and Text-alignment metrics.
3.6.2 Ablations Here, we ablate the critical design decisions for Movie Gen Video. For all ablations described in this section, we use a simpler, smaller baseline training and model setup than used for the main results. We analyze the effect of each design decision quantitatively via text-to-video human evaluation on a subset of Movie Gen Video Bench containing 381 prompts, termed Movie Gen Video Bench-Mini, and report results on text faithfulness and overall quality (see Section 3.5). For each ablation, every aspect of the model except for the design decision being tested is held constant for fair comparison. Next, we describe the simpler baseline setup followed by each ablation result. Unless described otherwise here, all other settings for the ablation experiments follow our 30B model including text encoders, flow matching objective, image training set, etc. Baseline model setup for ablations. We use a 5B parameter version of Movie Gen Video of trained to produce 352 ⇥ 192 videos of 4 – 8s. We use the TAE described in Section 3.1.1, which does 8⇥ compression across every spatio-temporal dimension to produce latents of shape 16 ⇥ 24 ⇥ 44. This smaller Movie Gen Video model has 32 layers in the transformer with 3072 embedding dimension and 24 heads. Baseline training setup for ablations. We use a two-stage training pipeline: (1) text-to-image pretraining; (2) text-to-image and text-to-video joint training. For simplicity, we used a smaller dataset of 21M videos, captioned with LLaMa3-Video 8B, that have a constant landscape aspect ratio for video training. First, we train the model on the image dataset with a learning rate of 0.0003, a global batch size of 9216 on 512 GPUs for 96K iterations. Next, we perform joint text-to-image and text-to-video training with an iteration ratio of 0.02 : 1 where the global batch size is 4096 for images and 256 for videos. We use a learning rate of 5e-5 and train for 100K iterations. Ablation Result - Training objective. We compare the Flow Matching training objective to the diffusion training objective. Following (Girdhar et al., 2024), we use the v-pred and zero terminal-SNR formulation of diffusion for training which is effective for video generation. As the human evaluation results in Table 8a show, Flow Matching leads to better generations both in terms of overall quality and text alignment while controlling for all other factors. Empirically, we also found this result to also hold across a range of model sizes and thus use Flow Matching to train our models.
Ablation Result - Effect of video captions. As described in 3.2.1, our video generation model is trained using clips from real videos and LLaMa3-Video generated video clip captions. To assess the importance of video captions, we compare our LLaMa3-Video 8B video captioning model to an image-based captioning scheme also based on LLaMa. This image-based captioning model captions the first, middle, and last frame of the video clip and then uses LLaMa to rewrite these three image-based captions into a single video caption. We refer to this model as LLaMa3-FramesRewrite. We first compare the quality of the two captioning schemes with human evaluations based A/B testing. Human raters are asked to pick between two given captions for the same clip. LLaMa3-Video generated captions are preferred 67% of the time while LLaMa3-FramesRewrite captions are only preferred 15% of the time. We visually observe that the video captioning model is able to accurately describe more fine-grained details regarding movements in the video. These fine-grained details provide a stronger supervision signal for training the video generation model, significantly improving overall prompt alignment by 10.8% (Table 8), with most of the increase coming from motion alignment (+10.7%) particularly on prompts that require ask for a high degree of motion in the output video (+16.1%). Ablation Result - Model architecture. In our work, we choose a transformer architecture based on LLaMa3 (see Section 3.1.3). We compare this to a Diffusion Transformer (Peebles and Xie, 2023) based model, which is commonly used in the media generation literature (Peebles and Xie, 2023; OpenAI, 2024; Ma et al., 2024a). The architecture differences between these two models can be seen in Table 9. As shown in Table 8c, we find that our LLaMa3 based architecture significantly outperforms the Diffusion Transformer on both quality (18.6%) and text-alignment (12.6%). This significant result shows that the LLaMa3 architecture has an advantage over the commonly used DiT for media generation. Our goal with Movie Gen is to scale to large model sizes, and to the best of our knowledge we find no detailed examples in the literature of scaling the Diffusion Transformer to very large scales. This result demonstrates that we can confidently transition from the commonly used Diffusion Transformer to architectures more commonly used in LLMs such as LLaMa3, the scaling behavior of which has been well documented (Touvron et al., 2023; Dubey et al., 2024)
3.6.3 TAE Results We present here results and ablations from important design decisions for the temporal autoencoder (TAE). For evaluation, we report the reconstructed peak signal-to-noise ratio (PSNR), structural similarity (SSIM) (Wang et al., 2004), and Fréchet Inception distance (FID) (Heusel et al., 2017) of video clips split from the training set, with 2s, 4s, 6s, and 8s duration, each with 200 examples. We also measure the same metrics on a validation split of the image training set. For video reconstruction evaluation, metrics are averaged over video frames. Qualitative Results. We show sample reconstructions from our TAE in Figure 16 with frames from the original video and the reconstruction after the TAE encoder and decoder. We observe that the TAE can reconstruct the video frames while preserving visual detail. The TAE reconstruction quality decreases for high frequency spatial details in images and video frames, and fast motion in videos. When both high frequency spatial details and large motion are present in a video, this can lead to a loss in detail, as can be seen in the examples in Figure 16, where fine details are smoothed out in the reconstruction. Quantitative metrics. Table 10 compares our TAE against a baseline frame-wise autoencoder that does not perform any temporal compression. Our baseline also produces a 8 channel latent space as is standard for autoencoders used for frame-wise encoding in prior work (Blattmann et al., 2023a; Girdhar et al., 2024). On video data, we observe that the TAE achieves a competitive performance to the frame-wise encoder while achieving a 8⇥ higher temporal compression. On images, the TAE outperforms the frame-wise model, an improvement that can be attributed to the increased channel size of the latent (8 vs. 16) (Dai et al., 2023).
Figure 16 Real (left) and TAE reconstructed (right) videos. The TAE compresses the video by a factor of 8⇥ across each of the three spatiotemporal dimensions. We observe that the reconstructions from the TAE maintain visual detail present in the original videos. Video (512 px) Image (512 px) SSIM " PSNR " FID # SSIM " PSNR " FID # Frame-wise AE 0.9348 34.11 0.9352 0.8877 30.83 1.7588 TAE 0.9093 32.25 1.4872 0.9231 32.16 0.2873 Table 10 TAE reconstruction metrics comparison between an frame-wise autoencoder and our TAE model. We observe that the TAE achieves comparable performance to a frame-wise autoencoder for video reconstruction while achieving a 8⇥ higher compression. 3.6.4 TAE Ablations We now perform a series of ablation experiments for the design choices in training our TAE model. Baseline setting for ablations. For simplicity, we use a TAE model with a smaller 4⇥ compression ratio that produces a 8-channel latent space. 2.5D vs. 3D attention & convolutions. We compare using 2.5D, i.e., 2D spatial attention/convolutions followed by 1D temporal attention/convolutions to using 3D spatiotemporal attention/convolutions in the TAE. In Table 11, we observe that the 3D spatiotemporal attention leads to slightly better reconstruction metrics. However, we found that this improvement was not large enough to justify the larger memory and compute costs associated with a fully 3D model compared to a 2.5D model. Thus, we use 2.5D for our TAE.
Table 11 Comparing TAE models with 2.5D vs. 3D convolutions and attentions. We observe that while the 3D models perform slightly better than the 2.5D models, the gap in performance between the two models is small. Effect of outlier penalty loss. We ablate the effect of adding the outlier penalty loss (OPL) from Section 3.1.1. The addition of this loss removes the artifacts from generated and reconstructed videos as seen in Figure 5, and improves reconstruction performance. We first train a baseline model without OPL for 50K iterations. We then finetune this model with OPL for 10K iterations and compare it to a baseline finetuned without OPL for 20K iterations. The results, summarized in Table 12, suggest that OPL finetuning improves the reconstruction for both images and videos. 3.6.5 Spatial Upsampler Results Here, we include some results from the spatial upsampler described in Section 3.1.5. A visual comparison of the upsampling process is presented in Figure 17, which shows the 200 px and 400 px crops before and after upsampling. The results demonstrate that the upsampler effectively sharpens and enhances visual details, producing a more refined and detailed output. 3.7 Text-to-Image Generation The Movie Gen model is trained jointly on videos and images, and hence is able to generate both videos and images. To further validate the model’s image generation capabilities, we continued training it with an image autoencoder and compared its performance to prior work in image generation. The following sections provide detailed experimental settings and evaluation results. 3.7.1 Method For the Text-to-Image model, our goal is to generate realistic images. We utilize the Movie Gen model as an initialization and replace the TAE with an image autoencoder. We then train the model on the text-to-image generation task, allowing it to generate images based on text descriptions. The final resolution is 1024 px. For post-training, we curated a total of O(1000) images created by in-house artists for quality-tuning, following the approach outlined in (Dai et al., 2023). We finetuned the model for 6k steps with a learning rate of 0.00001 and a batch size of 64. We used a constant learning rate scheduler with 2000 warm-up steps. 3.7.2 Results To measure the quality of our Text-to-Image generation results, we use human evaluators to evaluate the following axes: (a) text faithfulness, and (b) visual quality. For evaluating text faithfulness, we use a pairwise A/B comparison set up where evaluators select which image aligns better with a given generation prompt. Evaluators are asked to choose which of the choices A or B, is better, or equal, in terms of text alignment. For visual quality, we use a a similar pairwise A/B comparison set up and ask raters to help select the image that looks more realistic. Evaluators are asked to look for flaws in the generation, such as errors in the number of fingers or arms, or visual text spelling errors, before making their decision. For creating the benchmarking prompts, we analyzed typical text-to-image user prompts and generated categories and distributions, and leverage LLMs to produce user prompts that mimic real users prompts. We compare with the best contemporary Text-to-Image models including Flux.1 (Black Forest Labs, 2024), OpenAI Dall-E 3 (OpenAI, 2024), Midjourney V6.1 (Midjourney, 2024), and Ideogram V2 (Ideogram, 2024) available at time of benchmark These are however black-box commercial solutions, which makes fair comparison  a challenge. Similar to Text-to-Video evaluation, we obtain non-cherry picked generated images from the benchmark prompts for the prior work methods, and compare to them using non-cherry picked images from Movie Gen for the same prompts. To ensure consistent comparison across all models and evaluation axes, we utilize the ELO rating system to establish rankings based on battle records converted from raw human evaluation results. For A/B comparison evaluations, the “win/tie/lose” on a given prompt between two models were directly interpreted as one battle record. This approach allowed us to combine ratings on all evaluation axes to generate an overall performance. The comparison results are summarized in Figure 18, where we see that our model achieves the highest ELO rating compared to all recent state-of-the-art text-to-image methods available at the time of benchmarking. In Figure 19, we show some qualitative results of our generations.
Figure 18 ELO rating comparison of text-to-image methods. We compare our image generation model to state-of-the-art text-to-image models and observe that it performs competitively with recent approaches. Figure 19 Example T2I Images from Movie Gen. Generation prompts from left to right: (1) A focused writer, with a trusty pencil by their side, sits in a modern, minimalist home office with a large window providing ample natural light. (2) A hiker wearing a backpack walks along a trail with a rugby ball in hand, looking up at a soaring airplane overhead, the sun shining down on a beautiful day. (3) A styled hair with a headband inspired a graphic designer using a pen tablet and stylus to create vibrant digital art in a cubism style. (4) A vivid landscape painting on an easel stands in a bright, natural light-filled studio, showcasing its vibrant colors and textures, while a painter works
4 Video Personalization Generating personalized high quality videos that accurately capture an individual’s identity is an important research area with significant practical applications. We integrate personalization into video generation, yielding state-of-the-art outcomes as detailed in this section. We describe our novel model architecture in Section 4.1 followed by the training recipe in Section 4.2.1 and Section 4.3. We explain the evaluation criteria for personalization in Section 4.4 and show quantitative results in Section 4.5. 4.1 Model We extend our 30B Movie Gen Video model for Personalized Text-to-Video generation, PT2V, by conditioning the model on the identity information extracted from an input reference image in addition to the text prompt. Figure 20 illustrates the architecture of our PT2V model initialized from the T2V Movie Gen Video weights. We use vision token concatenation in the condition, enabling integration into a unified framework, which allows to scale up the model size. Similar to (He et al., 2024b), we extract identity features from a masked face image using a trainable Long-prompt MetaCLIP vision encoder (Xu et al., 2023), followed by a projection layer to align them with the text feature dimension. Our training strategy includes a PT2V pre-training phase followed by PT2V high quality finetuning.
Figure 20 Architecture and inference pipeline of the Personalized Movie Gen Video (PT2V) model. We initialize the model from Movie Gen Video’s weights and add additional learnable parameters to enable conditioning on a reference image. We encode the reference image using a trainable vision encoder initialized from Long-prompt MetaCLIP (Xu et al., 2023), and concatenate the embedding with the text prompt embedding. Frozen layers are illustrated in yellow while trainable ones in green.
4.2 Pre-training 4.2.1 Pre-training Data For the PT2V training sets, our focus is exclusively on videos where the same person appears across all frames. We curate this training set from the Movie Gen Video pre-training datasets described in Section 3.2.1. To achieve this, we first filter the raw T2V videos based on captions by selecting those with human-related concepts. We extract frames at one-second intervals and apply a face detector to keep videos that contain a single face and where the ArcFace cosine similarity score (Deng et al., 2019) between consecutive frames exceeds 0.5. This processing provides us with O(1)M text-video pairs where a single person appears, with durations from 4s to 16s. Based on the source reference face, our PT2V training dataset can be categorized into “paired” and “cross-paired” data. We define “paired” data as cases where the reference image is taken from the same video clip, while “cross-paired” data refers to cases where the reference image originates from a different video but features the same subject. Paired Data. For each selected text-video pair, we uniformly sample 5 frames from the video clip, yielding O(10)M paired training samples. For each frame, we crop the face area and segment the face region to prevent the model attending to non-critical areas such as the background. Cross-Paired Data. We observed that training solely on the above paired data makes the model easily learn a copy-paste shortcut solution, i.e., the generated video always follows the expression or the head pose from the reference face. To address this issue, we collect training pairs where the reference image comes from a different video of the same person. We collected both real and synthetic cross-paired data samples. O(10)K real cross-pairs from a subset of our pre-training data that contains different camera views of the same scene. For the synthetic cross-paired data, we use a pre-trained personalization image generation model (He et al., 2024b) to create synthetic reference images. Specifically, we apply the model to the first frame of each video from the paired data, generating images with diverse prompts to vary expressions, head poses, and lighting conditions, etc. To maintain identity consistency, we discard any generated images with an ArcFace similarity score below 0.7 compared to the reference image. In total, this process yields O(1)M synthetic cross-paired data samples. 4.2.2 Pre-training recipe There are three goals in PT2V pre-training: 1) train the model to condition on a reference image and preserve the identity, 2) generate long personalized videos, and 3) improve generated human expressions and motion naturalness. We found that directly training the model on long videos is inefficient and often leads to slow identity injection to the personalized model since (1) training speed is nearly proportional to the square of the number of latent frames (tokens), and (2) the weak reference image-to-video correspondence in long videos makes the task more challenging. More details on the pre-training recipe is shared in Figure 21. Stage-I: Identity injection. In the first stage of PT2V pre-training, we simplify the problem by conditioning the model on the reference image and training on short videos. Specifically, we truncate the TAE embedding to 8 latent frames (corresponding to 64 RGB video frames) to accelerate identity injection using the paired training samples. We freeze the vision encoder and only train the transformer backbone. We observe that the model can quickly learn to follow the reference image during this stage, as measured by the average ArcFace similarity score in Figure 21. Stage-II: Long video generation. To recover the model’s capability to generate long videos, we continue training the PT2V model from Stage-I with a larger number of latent frames, similar to the pre-trained T2V model in Table 2. This stage substantially enhances the consistency of long video generation, particularly in terms of background and motion coherence
Stage-III: Improve naturalness. Since the model in stage-I and stage-II has been trained on the paired imagevideo samples, it often demonstrates a strong copy-paste effect. For instance, in the generated video frames, the person tends to gaze directly at the camera, resulting in an unnatural-looking facial expression. We improve video naturalness and facial expression in stage-III by training on the cross-paired samples where the reference image is not from the corresponding target video. We leverage both real cross-paired data and 30 Training stage TP CP bs/Node #GPUs global bs learning rate frame length #iters #seen videos Stage-I 4 2 1 4096 512 2e-5 64 18k 9.21M Stage-II 4 2 1 4096 512 2e-5 128/192/256 8k 4.1M Stage-III 4 2 1 4096 512 2e-5 Transformer 128/192/256 4k 2.05M 1e-7 Vision Encoder Total 30k 15.36M Figure 21 Personalized Movie Gen Video (PT2V) pre-training. Left: PT2V pre-training recipe. Our training recipe consists of three stages as described in Section 4.2.2. Right: ArcFace similarity score of the PT2V Stage I-III pre-training. The model gradually follows the identity as the training going on in Stage-I. Stage-II maintains the identity similarity of Stage-I. In Stage-III, identity similarity tends to fluctuate due to training with cross-paired data. synthetic cross-paired data in this stage as discussed in Section 4.2.1. We also finetune the vision encoder to extract more detailed identity features from the reference image
4.3 Supervised Finetuning Similar to T2V, we further improve the video aesthetics in a high-quality finetuning stage by leveraging high quality aesthetic data. 4.3.1 Finetuning Dataset The large scale pre-training data enables the model to generate videos following the identity from the reference face image. Similar to the post-training of Movie Gen Video (see Section 3.3), we collect a small set of high-quality finetuning data, with the goal of generating highly aesthetic videos with good quality motion. To match the visual quality and aesthetics of Movie Gen Video, we started from the T2V finetuning set and collected videos with a single person. Subsequently, we manually selected videos with diverse human actions, ensuring that the dataset captured a variety of movements and behaviors. In total, our final finetuning set contains O(1000) high-quality videos with both paired and real cross-paired reference images used with a 1:1 ratio. 4.4 Evaluation We evaluate the quality of our PT2V models across three axes: identity preservation, video quality, and video-text alignment. The latter two axes are similar to T2V A/B evaluations in Section 3.5, where video quality can be further broken down into overall quality, frame consistency, motion completeness, and motion naturalness. To measure identity preservation, given an identity reference image and the generated video clip, the annotators are asked to rate on how well the generated character’s face captures the reference person likeness in both the best and the worst frame (identity score), as well as how visually consistent the faces are among the generated frames containing the reference person (face consistency score). These two scores are measured in an absolute sense with ratings as “really similar”, “somewhat similar”, and “not similar” for the identity question and “really consistent”, “somewhat consistent”, and “not consistent” for the face consistency question. Annotators were trained to follow specific guidelines for the labeling on these axes and are constantly audited for quality. Evaluation Dataset. We selected 50 subjects who were not seen during training as the reference face in the evaluation data. These reference face images include both frontal and side views. For each image, we pair it with 5-7 unique prompts, and curate 330 image-prompt pairs for evaluation. Similar to the T2V evaluation datasets, these prompts cover different human activities and facial expressions. We follow the same prompt rewrite as Section 3.4.1 to bridge the gap between our training and inference captions
Table 13 Personalized Movie Gen Video (PT2V) evaluation. We compare our model after the pre-training and supervised high-quality finetuning stages against ID-Animator (He et al., 2024a) on Identity score on the best similar frame, the worst similar frame, and face consistency across frames. Method Identitybest (") Identityworst (") Face Consistency (") Frozen Vision Encoder 73.93% 66.67% 83.50% Trainable Vision Encoder 90.10% 87.22% 99.69% Cross-Paired Training 71.79% 66.36% 97.53% Table 14 Ablation study for Personalized Movie Gen Video in terms of identity preservation metrics. We observed that the trainable vision encoder better preserves identity than the frozen vision encoder. Note that although cross-paired training decrease the identity similarity, it leads to more diverse head poses and more natural expressions. PT2V-Finetune net win rate vs. ID-Animator Overall Quality 64.74 Consistency 22.18 Motion Naturalness 37.38 Motion Completeness 5.17 Text Alignment 53.20 (a) PT2V-Pretrain net win rate vs. T2V-Pretrain 3.95 10.33 -1.82 -5.16 -11.25 (b) Table 15 Personalized Movie Gen Video (PT2V) evaluation on video quality and text alignment. (a) Net win rate (win% - loss%) of our PT2V after supervised finetuning vs. SOTA (ID-Animator (He et al., 2024a)). PT2V significantly outperforms ID-Animator in all metrics. (b) PT2V vs. Movie Gen Video (T2V) without the visual conditioning. We observed that PT2V wins consistency and performs on par in overal quality accounting for statistical significance, but loses in motion completeness and prompt alignment due to the narrow concept distribution (activities, objects, etc.) of PT2V.
4.5 Results In Table 13 and Table 15a, we compare our Personalized Movie Gen Video after supervised finetuning with ID-Animator (He et al., 2024a). For the Identity score, we aggregate the “really similar” and “somewhat similar” scores in the best frame, and for the consistency score, we aggregate the “really consistent” and “somewhat consistent” scores. As evident, our method significantly outperforms the baseline by a large margin in all axes of identity preservation, video quality, and text alignment. We also compare it with Movie Gen Video without the visual conditioning in terms of video quality and text alignment in Table 15b. We present generated videos from Personalized Movie Gen Video in Figure 23. The first four videos are generated with the same prompt but different identities, and the latter four are generated with the same identity but different prompts. The generated videos follow the identity with diverse motion and camera views. Qualitative comparisons between Personalized Movie Gen Video and ID-Animator (He et al., 2024a) are shown in Figure 22. Personalized Movie Gen Video consistently outperforms ID-Animator in terms of identity consistency and video quality. 4.5.1 Ablations We ablate the impact of key design choices in our 30B Personalized Text-to-Video training pipeline. Effect of training visual conditioning embedding. Our models use an embedding from a visual encoder of the face as the visual embedding to condition the generation. We study whether training this embedding jointly during the video generation task improves performance. We re-train the third stage of our model with either a fixed or trainable vision encoder model and report the evaluation results in Tables 14 and 16. We observe that using a fixed vision encoder compromises identity preservation significantly, 16% as seen in Table 14. Reference Image Prompt: A person dressed in elegant attire is seen checking the table settings Table 16 Ablation study for Personalized Movie Gen Video in terms of Text-to-Video evaluation metrics. Each column shows the net win rate on adopting a design decision vs. a model that does not have it. Using a trainable vision encoder is comparable to the frozen one in terms of quality and alignment metrics while boosting the identity preservation significantly as in Table 14. The results also confirm the importance of cross-paired training data and supervised finetuning for video naturalness, higher quality and text alignment. Effect of cross-paired data. Our training pipeline uses cross-paired data, i.e., where the image of the face used to condition the generation comes from a different video clip than the video clip to be generated. We observe in Table 14 that cross-paired training leads to a decrease in identity metrics, however, it is crucial in improving facial expressions and natural movement in the generated videos. Human annotation in Table 16 reveals that cross-pair trained model improves text alignment by 27.36% and overall quality by 13.68%, especially 26.14% in motion naturalness. Effect of high quality finetuning. We show the impact of a final high-quality finetuning stage on all axes of video quality and text alignment in Table 16 and on identity preservation in Table 13. Similarly, since our high-quality finetuning set includes cross-paired data, identity drops slightly while video quality and naturalness is improved significantly.