MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences

## Abstract

Abstract Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: https://aim-uofa.github.io/MovieDreamer/ .

## 1 Introduction

Driven by the advent of generative modeling techniques (Rombach et al., ; Podell et al., ), there has been significant progress in video generation in the research community (Guo et al., ; Ho et al., ; Blattmann et al., ). Tremendous research efforts have focused on adapting a pre-trained text-to-image diffusion model (Guo et al., ; Ho et al., ) to a video generation model. Recently revolutionary leap has been made with the Sora (Brooks et al., ) model which substantially scales a spatial-temporal transformer model. This model demonstrates remarkable video generation quality, dramatically extending the duration of generated videos from seconds to an unprecedented full minute. This technology profoundly reshapes the boundaries of generative AI, fostering anticipation towards hours-long movie production.

While predominant video generation methods adopt diffusion models, this paradigm is more suited for visual rendering and is less adept at modeling complex abstract logic and reasoning compared to autoregressive models, as evidenced in natural language processing. Moreover, diffusion models lack the flexibility to support arbitrary length. On the other hand, autoregressive models (Wang et al., ; Jiang et al., ; Hu et al., ; Pan et al., ) have shown superior ability in handling complex reasoning and are better at predicting what might happen in the next—a key element of a world model. Moreover, autoregressive models offer greater flexibility in handling varied lengths of predictions and can also benefit from mature training and inference infrastructure. As such, there are attempts that use the autoregressive model for video generation (Kondratyuk et al., ; Hu et al., ). However, autoregressive modeling is not as compute efficient as diffusion model for visual rendering and demands substantially more computing even for image rendering.

Figure: Figure 1: The Titanic movie scenes generated by the proposed MovieDreamer. The generation is conditioned on the elaborate multimodal movie script. *The original movie is unseen during training.* Please refer to Figure 10, 11, 12, 13, 14 for more results.
Refer to caption: extracted/5912847/images/Teaser0/1.jpg

Recent progress (Henschel et al., ; Chen et al., ; Qiu et al., ; Brooks et al., ) suggests that video generation quality in short duration (from seconds to a minute) can be consistently improved by scaling the compute due to the scaling law. In this work, we sidestep the short video generation problem and instead seek a generative framework for long video generation featuring complex narrative structures and intricate plot progressions, which is hardly achieved by merely scaling the current approaches. Our aim is to push the boundaries of video generation beyond the limitations of short-duration content, enabling the creation of videos with rich, engaging storylines.

In this paper, we introduce a hierarchical approach, dubbed *MovieDreamer*, that marries autoregressive modeling with diffusion-based rendering to achieve a novel synthesis of long-term coherence and short-term fidelity in visual storytelling. Our method leverages the strengths of autoregressive models to ensure global consistency in key movie elements like character identities and cinematic styles, even when the camera switches back and forth. The model predicts visual tokens for specific local timespans, which are then decoded into keyframes and dynamically rendered into video sequences using diffusion rendering. Our method mirrors the traditional movie directing process, where intricate plots are decomposed into distinct, manageable scenes. Our method ensures that both the overarching narrative and the minute spatial-temporal details are maintained with high fidelity, thus enabling the automated production of movies that are both visually cohesive and contextually rich.

To this end, we represent the whole movie using sparse keyframes and employ a diffusion autoencoder to tokenize each keyframe into compact visual tokens. We train an autoregressive model to predict the sequence of visual tokens. This model is initialized from a pretrained large language mode, a 7B LLaMA model (Touvron et al., ), to leverage its rich world knowledge for better generalizability. As such, the model can be regarded as a multimodal model, which, rather than outputting the text, predicts the visual tokens with movie script conditioning.

Three key designs need to be specially considered during the autoregressive training. One is to resort to various model and input perturbation techniques to combat the overfitting tendency since the high-quality movie training data is limited. Besides, we propose to condition the model using a novel *multimodal script*, which, for each keyframe, includes the rich description of the image style and scene elements, along with detailed text description of characters as well as their retrieved face embeddings. Such multimodal script facilitates narrative continuity across different segments of the movie while offering flexibility for character control. Moreover, we stochastically append a few reference frames of the same episode at the beginning of the input, and in this way, the model acquires in-context learning capability and can synthesize higher-quality personalized results. Such feature is particularly useful in generating coherent continuations of given movie episodes.

We subsequently decode the predicted vision tokens using an image diffusion decoder and further render the image to the video clip using an image-to-video diffusion model. Importantly, we enhance this process by finetuning an *identity-preserving* diffusion decoder. This refinement effectively improves identity preservation in the resultant video clips. Our work is essentially orthogonal to the current efforts aimed at improving the short video generation quality via compute scaling and our method can benefit from these progresses. Our approach is validated through extensive experiments on a wide range of movie genres, showcasing an excelling ability to generate visually stunning and coherent long-form videos over the state-of-the-art.

To summarize, the contributions of this paper are:

- •
We introduce MovieDreamer, a novel hierarchical framework that marries autoregressive models with diffusion rendering to generate long-term coherent stories and videos with multiple characters well-preserved. The method substantially extends the duration of generated video content to thousands of keyframes. *To the best of our knowledge, we are the first to tackle the ultra-long narrative video generation problem, rather than merely clip looping.*
- •
We generate the visual token sequence using a multimodal autoregressive model. The autoregressive model supports zero-shot and few-shot personalized generation scenarios and supports variable length of keyframe prediction.
- •
We use a novel multimodal script to hierarchically structure rich descriptions of scenes, plots, as well as the character’s identity. This approach not only facilitates narrative continuity across different segments of a video but also enhances character control and identity preservation.
- •
Our method demonstrates superior generation quality
with detailed visual continuity, high-fidelity visual details, and the character’s identity-preserving ability. Both story generation and video generation can be addressed with our method.

## 2 Related Work

#### Video generation models.

Video generation has advanced mainly through diffusion and autoregression methods. The popularization of Stable Diffusion (Rombach et al., ) has sparked extensive studies, improving video quality with various techniques (Guo et al., ; Ho et al., ; ; He et al., ). For long videos, strategies like (Yin et al., ; Qiu et al., ; Wang et al., ; Chen et al., ; Henschel et al., ) can only generate simple results. Despite their high computational load and challenges in managing complex sequences, diffusion models are widely used. Autoregression, on the other hand, is less common but excels in specific areas like autonomous driving by employing unified token spaces (Hu et al., ), although it struggles with continuity in complex scenes. Our approach combines the strengths of both methods in a hierarchical and scalable manner, enhancing control and flexibility irrespective of video length.

#### Multimodal large language models.

Recent advancements in NLP lead to the emergence of visual language models (VLMs) through enabling large language model (Touvron et al., ) to comprehend visual modalities (Liu et al., ; Bai et al., ; ; Chen et al., ). In addition to language, VLMs have demonstrated their power in generating images (Pan et al., ; Sun et al., ; ). Moreover, ongoing research (Zhang et al., ; Lin et al., ; Li et al., ; Zhao et al., ) has showcased the robust capabilities of Multimodal Language Models (MLLMs) in comprehending video content. Some works (Cai et al., ; Cheng et al., ; Ren et al., ; Rasheed et al., ; Lai et al., ) enable VLMs to assist in spatial understanding and other visual tasks. VLMs (Jiang et al., ; Kim et al., ; Etukuru et al., ; Brohan et al., ; Driess et al., ) also demonstrate the powerful capabilities in robotics. These advancements have solidified the effectiveness of VLMs in addressing complex problems. Building on this foundation, our approach leverages VLMs to facilitate the generation of extremely long video content.

#### Visual story generation.

The crux of visual story generation lies in ensuring consistency across generated images. Similar to video generation, diffusion and autoregression are also two primary manners. Existing diffusion-based methods predominantly employ conditional generation to maintain the coherence  (Su et al., ; Wang et al., ; Tewel et al., ; Hertz et al., ). Some (Li et al., ; Wu et al., ) have shown promising results in retaining high-quality facial features and identity.
Other approaches are based on the autoregressive ideas  (Feng et al., ; Liu et al., ; Rahman et al., ; Pan et al., ). However, these approaches generally yield mediocre results and are challenging to scale up. Story and video generation share intrinsic links, with overlapping solutions. Our method uniquely integrates these, offering a unified approach.

## 3 Method

### 3.1 Overview

Figure: Figure 2: The framework of our MovieDreamer. Our autoregressive model takes multimodal scripts as input and predicts the tokens for keyframes. These tokens are then rendered into images, forming anchor frames for extended video generation. Our method ensures long-term coherence and short-term fidelity in visual storytelling with the character’s identity well preserved.
Refer to caption: x1.png

We propose a novel framework for generating extended video sequences that leverages the strengths of autoregressive models for long-term temporal consistency and diffusion models for high-quality image and video rendering.
Our method takes multimodal scripts as conditions, predicts keyframe tokens in an autoregressive manner, renders the tokens into images and uses these images as anchors to produce full-length videos. Our method offers flexibility to support zero-shot generation along with few-shot scenarios in which the generation results are required to follow the given reference. We take special care of preserving the identity of the characters throughout the multimodal script design, autoregressive training, and diffusion rendering.
We illustrate the overall framework in Figure 2.

### 3.2 Keyframe Tokenization via Diffusion Autoencoder

To create a concise yet faithful representation of images, we employ a diffusion autoencoder. Our encoder, $\mathcal{E}$, contains a pretrained CLIP vision model (Radford et al., ) and a transformer-based token compressor, which encodes an image $\mathbf{x}$ into a reduced number of compressed tokens denoted by $\mathbf{e}$. The decoder, $\mathcal{D}$, is finetuned based on the pretrained SDXL and yields an 896$\times$512 reconstructed image by leveraging the compressed tokens in the cross-attention module.
We train the compressor and the decoder while leaving the CLIP vision model frozen. The loss for training this diffusion autoencoder is as follows:

$$ $\mathcal{L}_{DAE}=\mathbb{E}_{\mathbf{x}_{0},\epsilon\sim\mathcal{N}(0,\mathbf {I})}\|\epsilon-\mathcal{D}(\mathbf{z}_{t},t,\mathcal{E}(\mathbf{x}_{0}))\|^{2 }_{2},$ (1) $$

where $\mathbf{x}_{0}$ is the input image and $\mathbf{z}_{t}=\alpha_{t}\mathbf{x}+\sigma_{t}\bm{\epsilon}$ is its corresponding noisy latent at timestep $t$. $\bm{\epsilon}\in\mathcal{N}(0,I)$ is the Gaussian noise. $\alpha_{t}$
and $\sigma_{t}$
define the noise schedule. In our experiment, we empirically find that merely two tokens can sufficiently characterize major semantics of keyframes, corroborating the finding of previous work (Yang et al., ).

### 3.3 ID-preserving Diffusion Rendering

In our approach, while the primary diffusion decoder $\mathcal{D}$ adeptly reconstructs target images, it occasionally falls short in capturing fine-grained details, notably in facial features, due to the attenuation of details in compressed tokens. To address this, we enhance the cross-attention module within $\mathcal{D}$. This enhancement involves the integration of both descriptive text embedding, $\mathbf{d}$, and face embedding, $\mathbf{f}$, derived from the multimodal script. Specifically, the face embedding and description embedding is projected to the same dimension as the compressed token with two additional MLPs, which are then concatenated together and serve as the input to the keys and values of cross-attention modules in $\mathcal{D}$.

To further advocate the model’s ability to focus on critical details, we introduce a random masking strategy that obscures a subset of the input tokens. This technique encourages the decoder to more effectively utilize the available facial and textual cues to reconstruct the images with higher fidelity, particularly in preserving identity-specific characteristics. This ID-preserving rendering also compensates for the loss of identity during the autoregressive modeling, leading to substantially improved identity perception quality as illustrated in Figure 3.

### 3.4 Autoregressive Keyframe Token Generation

We initiate our approach by leveraging a pretrained large language model, specifically LLaMA2-7B (Touvron et al., ), to construct our autoregressive model, $\mathcal{G}$. Unlike traditional LLMs, $\mathcal{G}$ is designed to predict compressed vision tokens from the multimodal script $\mathbf{M}$ of the corresponding keyframe and historical information $\mathbf{H}$, which is formulated as $\mathbf{e}_{t}=\mathcal{G}(\mathbf{H}_{<t},\mathbf{M}_{t})$.

Traditional LLMs often employ cross-entropy loss for training, which is suitable for discrete outputs. However, our model deals with continuous real-valued image tokens, which makes cross-entropy inapplicable.
Inspired by  Tschannen et al. (), we adopt a $k$-mixture Gaussian Mixture Model (GMM) to effectively model the distribution of these real-valued tokens. This involves parameterizing the GMM with $2kd$ means, $2kd$ variances, and $k$ mixing coefficients.
These parameters are predicted by attaching an additional trainable linear layer at the end of the autoregressive model, enabling the construction of the GMM and the sampling of continuous tokens from the GMM. The model is trained by minimizing the negative log-likelihood:

$$ $\displaystyle\mathcal{L}_{nll}$ $\displaystyle=\mathbb{E}(-\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{L} \operatorname{log}p(e_{t,i}|\mathbf{H}_{<t},\mathbf{M}_{t})),$ (2) $$

where $T$ represents the total number of frames, $t$ indexes the current frame, and $L$ is the number of compressed tokens per frame. To ease the learning for these continuous tokens, we also incorporate $\ell_{1}$ and $\ell_{2}$ losses between the predicted and ground truth token and optimize the following objective:

$$ $\displaystyle\mathcal{L}_{ar}=\mathcal{L}_{nll}+\ell_{1}(\mathbf{e}_{pred}, \mathbf{e}_{gt})+\ell_{2}(\mathbf{e}_{pred},\mathbf{e}_{gt}).$ (3) $$

#### Anti-overfitting.

It is challenging to massively acquire large amounts of high-quality, long-duration video data. We managed to collect 50k long videos, yielding 5M keyframes for autoregressive training. Yet this data volume is relatively limited, leading to a severe overfitting issue where the generative model fails to generalize well during testing. To address this, we implement several strategies which are crucial for training:

- •
Data augmentation. To maximally utilize our training data, we apply random horizontal flips and stochastically reverse the temporal order of video frames. Such training data augmentation considerably increases the diversity of training data.
- •
Face embedding randomization. To prevent identity leakage, we randomly retrieve face embeddings of the same character from different frames. Otherwise, the model will memorize the training frame simply from the face embedding input.
- •
Aggressive dropout. An unusually high dropout rate of 50% is utilized, which is crucial for generalized learning from limited training data.
- •
Token masking. We incorporate random masking of input tokens with a probability of 0.15, which is applied to the causal attention mask. This forces the model to infer missing information based on the available context (face ID), further strengthening its ability to generalize from partial data.

#### Multi-modal scripts for autoregressive conditioning.

We develop a well-structured multi-modal script format to serve as input for autoregressive models, as depicted in Figure 16. Our script integrates multiple dimensions: characters, scene elements, and plots. Accurately representing character appearance using text alone is challenging; therefore, we combine textual descriptions with facial embeddings (Zheng et al., ) to provide a more detailed representation of each character. To facilitate the processing by the autoregressive model, we structure the script format to distinctly delineate these elements.

For non-textual modalities such as facial embeddings and compressed tokens are projected to LLaMA’s embedding space using multi-layer perceptron. The primary challenge arises with textual data, which tends to produce long sequences, thereby consuming excessive token space and limiting the model’s contextual breadth. This challenge poses a significant obstacle to achieving our goal of *long* content generation. To mitigate this, we treat text as a separate modality, dividing it into “identifiers” and “descriptions” (elaborated in Appendix A). Identifiers establish the script’s structure, while descriptions detail the attributes for generation. Descriptions are segmented into sentences, with each encoded into a single [CLS] token using CLIP and then projected into the unified input space.
In other words, we perform the sentence-level tokenization.

This method significantly extends the usable contextual length during training by condensing an entire sentence into a single token. We utilize LongCLIP (Zhang et al., ) as our text encoder for descriptions, supporting inputs up to 248 tokens, which enhances our capacity to handle detailed narrative content. Consequently, the multi-modal script at timestep $t$ and preceding historical information are represented as:

$$ $\displaystyle\mathbf{M}_{t}$ $\displaystyle=\left[\mathbf{d}_{t},\mathbf{i}_{t},\mathbf{c}_{t}\right],\ \mathbf{H}_{<t}=\left[\mathbf{d}_{<t},\mathbf{i}_{<t},\mathbf{c}_{<t},\mathbf{ e}_{<t}\right],$ (4) $$

where $\mathbf{d}_{t}$, $\mathbf{i}_{t}$, and $\mathbf{c}_{t}$ denote the embeddings for descriptions, identifiers, and characters’ facial embeddings, respectively. $\mathbf{e}_{<t}$ represents the previously predicted compressed frame tokens. The negative log-likelihood loss in Equation 2 is then formulated as:

$$ $L_{nll}=\mathbb{E}(-\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{L}\operatorname{ log}p(e_{t,i}|e_{t,<i},\mathbf{e}_{<t},\mathbf{c}_{\leq t},\mathbf{d}_{\leq t} ,\mathbf{i}_{\leq t})).$ (5) $$

#### Few-shot training for personalized generation.

To promote personalized movie content generation, we propose a few-shot learning method that utilizes in-context learning. During training, we select 10 random frames from an episode, encode them into visual tokens, and prepend these tokens stochastically to the episode’s visual tokens. This strategy not only promotes in-context learning, allowing the model to tailor content based on the reference frames, but also acts as a data augmentation technique, effectively mitigating overfitting.

Our model is versatile, supporting both zero-shot and few-shot generation modes. In zero-shot mode, the model generates content from the multimodal script. In few-shot mode, it leverages a small set of user-provided reference images to align the generated content more closely with user preferences, without necessitating further training. This capability ensures that users can efficiently produce high-quality, customized visual content that aligns better with their desired target.

### 3.5 Keyframe-based Video Generation

After obtaining the keyframes, we can generate video clips of the movie based on these keyframes.
A straightforward approach is to utilize an existing image-to-video model, such as Stable Video Diffusion (SVD) (Blattmann et al., ), to generate these clips.
Specifically, SVD transforms the input image into latent features for conditioning and introduces interaction with the CLIP features of the input image via cross-attention.
Although SVD is capable of generating high-quality short videos, e.g., 25 frames, it struggles to generate longer movie clips.

To generate longer movie clips, a straightforward way is to utilize the last frame of the previously generated video as the initial frame for generating the subsequent video. This process can be iterated to obtain a lengthy video sequence.
However, we empirically found that this causes a serious accumulation of errors: the quality of the video frames gradually deteriorates as the time gets longer.

To tackle this, we propose a simple and effective solution. Our motivation is to always use the feature of the first frame as an “anchor” during video extension, to enhance the model’s awareness of the original image distribution.
In practice, we use the CLIP feature of the original input image, instead of the last frame of the previous video, for cross-attn interaction when generating subsequent videos.

*We want to emphasize that our key contribution lies in the hierarchical controllable very long video generation, with MLLM for keyframe generation and diffusion model for video rendering.*
Our subsequent experiments also demonstrate that existing high-quality video generation models trained by resource-rich institutions can be integrated with our MovieDreamer, producing better visually appealing results. This further demonstrates our flexibility.

Figure: Figure 3: The ID-preserving renderer significantly enhances the perceived identity of the character. For the second row, the input is only the compressed token. For the third row, the input is the compressed token, face ID, and description embedding.
Refer to caption: extracted/5912847/images/ab_postprocess/17_gt.jpeg

## 4 Experiments

### 4.1 Implementation Details

We use pretrained models to reduce the training computational overhead, which is very common in the era of large models. We use the U-Net of pretrained SDXL (Podell et al., ) as our diffusion decoder. The encoder consists of a pretrained CLIP image encoder (ViT-L) (Radford et al., ) and a transformer-based compressor. The autoregressive model is initialized with a pretrained LLaMA-7B (Touvron et al., ) checkpoint. All MLPs used to align different modal inputs follow the architecture in LLaVA (Liu et al., ).
Please refer to Appendix A for more details about the network architecture and training.

### 4.2 Experimental Setup

#### Dataset.

The majority of our training data is sourced from Condensed Movies (Bain et al., ), with additional data collected using the same methodology as this dataset, resulting in 5M keyframes with corresponding script annotations. To systematically evaluate the effectiveness of our method, we construct a test dataset consisting of 100 long movies that are NOT included in the training set, with 1M keyframes after pre-processing.
Each of these videos is processed through our proposed data pre-processing pipeline (see Appendix C), which accurately extracts and annotates keyframes.

#### Evaluation metrics.

We evaluate the effectiveness of various methods using the following metrics. The CLIP score (Radford et al., ) assesses semantic alignment between the plot and generated outputs. Quality is evaluated using the Aesthetic Score (AS) (Schuhmann et al., ), Fréchet Image Distance (FID) (Heusel et al., ), and Inception Score (IS) (Salimans et al., ). For assessing consistency, we propose new metrics for short-term (ST) and long-term (LT) consistency. ST is measured by calculating the CLIP cosine similarity between adjacent frames containing the target character. LT consistency is evaluated by checking the presence of the target character in selected frames across the test set. Finally, we calculate the ratio of frames containing the target character to the total number of selected frames as a measure of the accuracy in maintaining long-term character consistency. Please refer to the Appendix B for more details about the evaluation metrics.

Figure: Figure 4: Qualitative comparisons of story generation. Please refer to the appendix E for more comparisons. Each keyframe is generated with the corresponding multimodal script as input.
Refer to caption: extracted/5912847/images/compare_story/ours/77.jpeg

### 4.3 Comparison with the State-of-the-Art

#### Story generation.

Many existing story generation methods focus on fine-tuning with small datasets, exhibiting poor generalization.
Consequently, we compare our method only with those that demonstrate high generalization capabilities, namely StoryDiffusion (Zhou et al., ) and StoryGen (Liu et al., ).
As shown in Figure 4, StoryDiffusion consumes significant computational overhead and tends to generate
generic faces that do not align with the desired character.
Similarly, StoryGen fails to preserve consistency and generates abnormal results.
In contrast,
our method achieves generating extremely long content while preserving both short-term and long-term consistency across multiple characters.
The observation is also evidenced by the quantitative results in Table 1, where our method achieves high scores in both LT and ST.
Moreover, the higher CLIP score reflects that our generated results align well with the storyline.
Better IS, AS, and FID scores demonstrate our method achieves high-quality images.
We showcase more qualitative results in Appendix E.

Figure: Figure 5: Ablations on face embedding. Character consistency is well-preserved with face embedding. The autoregressive model generates compressed tokens with multimodal script as input. The results are rendered by the decoder $\mathcal{D}$, with compressed tokens as well as face embedding and description embedding as input.
Refer to caption: extracted/5912847/images/ab_face_embedding/Picture1.jpg

**Table 1: Quantitative comparisons of different methods. ST and LT refer to our proposed short-term and long-term consistency metrics, respectively. noC denotes without continuous token supervision.**
| Method | CLIP $\uparrow$ | Inception$\uparrow$ | Aesthetic$\uparrow$ | FID$\downarrow$ | ST$\uparrow$ | LT$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| StoryDiffusion | 15.232 | 8.739 | 6.134 | 4.643 | 0.627 | 0.596 |
| StoryGen | 13.261 | 6.216 | 3.837 | 8.557 | 0.508 | 0.542 |
| Ours | 19.584 | 9.698 | 6.093 | 2.043 | 0.646 | 0.814 |
| Ours-ref | 20.071 | 9.842 | 6.288 | 1.912 | 0.701 | 0.893 |
| Ours-noC | 18.833 | 8.609 | 5.893 | 2.714 | 0.606 | 0.755 |
| Ours-noC-ref | 19.431 | 8.774 | 5.979 | 2.560 | 0.616 | 0.776 |

#### Video results.

We perform a detailed comparison between our method and existing methods for generating long videos.
For text-to-video methods, we use detailed descriptions prepared in the test set as input.
For image-to-video approaches, we employ keyframes generated by our methods as inputs.
As illustrated in Table 2 and Figure 6 (b), our method significantly outperforms prior open-source models in terms of quality, demonstrating strong generalization capabilities. Most importantly, our method is capable of generating videos that last *tens of minutes or even hours* with little compromise in quality, achieving state-of-the-art results.
Here, we compare existing methods with our enhanced video model. As shown in Appendix D, when integrated with enterprise-level closed-source video models, our approach demonstrates even better results.

### 4.4 Analysis

#### Continuous token supervision.

We noticed a significant decline in the model’s ability to maintain ID consistency on the test set when using only L1 and L2 supervision. We believe this is because L1 and L2 losses are not good at modeling probability distributions, as they are more suitable for regression tasks. However, we need to model the complex conditional distribution of the autoregressive process. Therefore, we applied more refined supervision on continuous real-valued tokens. As illustrated in Figure 6, with continuous token supervision, the model is better capable of preserving target ID. Both the quality of the generated image and the preservation of character ID degenerate without continuous token supervision.

Figure: Figure 6: (a) With continuous token supervision, our model demonstrates stronger generalization capabilities and achieves higher-quality results. Better character ID is preserved with continuous token supervision. (b) Visualization of long video generated by different methods.
Refer to caption: x2.png

**Table 2: Quantitative comparison on full-length video generation.**
| Method | CLIP$\uparrow$ | Inception$\uparrow$ | Aesthetic$\uparrow$ | CLIP-sim$\uparrow$ |
| --- | --- | --- | --- | --- |
| StreamingT2V | 15.304 | 7.636 | 4.119 | 0.624 |
| SEINE | 19.404 | 7.463 | 5.825 | 0.688 |
| FreeNoise | 14.327 | 6.532 | 3.194 | 0.612 |
| Ours | 19.520 | 8.642 | 6.049 | 0.704 |

#### Anti-overfitting strategies.

Large autoregressive models are powerful learners, making it easy for them to overfit the dataset.
As shown in the first row in Figure 7, the generated contents are dominated by the input character. The model produces similar visual content even when given different text prompts.
Our anti-overfitting strategies are designed to weaken the correspondence between character IDs and target frames, thus avoiding simple memorization.
As can be seen in the second row, it helps produce diverse high-quality results that align well with the text description.

Figure: Figure 7: After resolving the overfitting, our model is capable of generating more diverse results. Each keyframe is generated with the corresponding multimodal script as input.
Refer to caption: extracted/5912847/images/overfitting/1.jpeg

#### Multi-modal movie scripts.

The multi-modal script introduces face embedding to better preserve consistency. Figure 5 compellingly demonstrates the effectiveness of this design. Specifically, the removal of face embeddings leads to a significantly decreased ability of the model to preserve character consistency. Face embeddings carry more nuanced and precise information compared to text alone.
With face embedding, both the short-term and long-term consistency are well-preserved.

#### ID-preserving renderding.

Before enabling ID-preserving rendering, our decoder has already shown the ability to reconstruct the target image. However, for images outside the training set, the reconstructed characters’ appearances may differ slightly from the expected targets due to the loss of fine facial features in the compressed tokens. After applying ID-preserving rendering, our decoder exhibits a significantly enhanced ability to preserve character identities. The experimental results, illustrated in Figure 3, clearly demonstrate the effectiveness of the post-processing step.

#### Few-shot personalized generation.

Our method serves as a powerful in-context learner, capable of generating results that are consistent with the style or characters of the few references provided by the user. The results are presented in Figure 8. Our model can produce results that are more consistent with the style and characters in the few-shot scenario.

## 5 Conclusion

We present MovieDreamer to address the challenge of generating long-duration visual content with complex narratives. This method elegantly marries the advantages of autoregression and diffusion, and is capable of generating very long videos. Additionally, we design the multi-modal script which aims at preserving character consistency across generated sequences.
We further introduce ID-preserving rendering to better preserve character IDs and support few-shot movie creation due to the in-context modeling. This work potentially opens up exciting possibilities for future advancements in automated long-duration video production.

Figure: Figure 8: Our zero-shot generation already achieves high success rates and quality. However, it occasionally struggles to preserve ID consistency accurately. Few-shot generation can produce results with better ID preservation.
Refer to caption: extracted/5912847/images/reference_new/11.jpg

## Appendix A Implementation Details

#### Diffusion Autoencoder.

Figure: Figure 9: Architecture of the compressor.
Refer to caption: x3.png

We use the pretrained CLIP image encoder (Radford et al., ) to generate spatial tokens. The spatial tokens are compressed into two tokens using the compressor, a network with 6 Transformer blocks. The architecture of the compressor is shown in Figure 9. First, we process the spatial tokens using three transformer blocks. We then interchange the dimensions of the hidden states and compress them using a linear layer. The compressed hidden states are further processed through three transformers to obtain the final compressed token. We find that setting the compressed token to 2 is sufficient for well reconstructing the image.

Subsequently, we employ the pretrained U-Net of SDXL (Podell et al., ) as our decoder to render the compressed tokens back into images. Notably, the original SDXL architecture demands an additional projected text embedding as input. To address this, we concatenate and project the compressed token into the desired input space of SDXL simply using a linear layer.

Compared to the image dataset, our collected movie dataset may lack diversity in all kinds of objects in the world, which somewhat limits the model’s performance. To ensure the model develops a general understanding of various objects first, we employed a three-stage training strategy to train a more effective model. In the first stage, we fine-tuned the pre-trained U-Net and the compressor on the large existing image dataset at a resolution of 768 x 768 with a cosine scheduling learning rate that ranges from 1e-4 to 2e-5 and an AdamW optimizer for 120k steps, allowing the model to understand various objects. Our dataset comprises images from OpenImages (Kuznetsova et al., ), JourneyDB (Sun et al., ), and Object365 (Shao et al., ).

In the second stage, we further fine-tuned the model using our self-collected movie dataset at a resolution of 896 x 512. The model is trained using AdamW with a constant learning rate of 2e-5 for 120k steps in the second stage. For the first stage and second stage, we first freeze the decoder and only train the compressor for the first 20k steps, driving it to compress images more appropriately.
To enable the model to generate results consistent with movie lighting, we further set the noise offset to 0.05 during training.

In the third stage, to better preserve character ID, we freeze the compressor and concatenate the compressed tokens with the text embedding and face embedding as inputs to the cross-attention and further finetune the decoder. Text embedding and face embedding are available in multimodal script data. Since the dimensions of the description embedding and facial embedding differ from those of the compressed token, we first project them to the same dimension as the compressed token using 2 two-layer MLPs with GELU as the activation function. Given that different frames have varying numbers of descriptions and face IDs, we pad the projected description embedding and projected facial embedding with zero tensors to extend the sequence length to 15 and 5, respectively. For instance, if a frame has a projected description embedding of shape [12, D] and a projected facial embedding of shape [3, D], we pad them with zero tensors of shape [3, D] and [2, D] to achieve shapes of [15, D] and [5, D]. Finally, we concatenate the compressed tokens with the projected description embedding and projected facial embedding, resulting in a [22, D] tensor, which serves as the key and value input for the UNet cross-attention block.
The decoder and the MLPs are trained using AdamW with a constant learning rate of 2e-5 for 120k steps and a noise offset of 0.05.
The entire training process of the compressor and decoder takes 3 weeks with 6 NVIDIA A800 GPUs.

#### Autoregressive Model.

We utilize a pretrained LLaMA 7B model as the backbone for our autoregressive model, which is subsequently trained with our curated multi-modal script data. We use two-layer MLPs with GELU as the activation function to unify different modalities into the input space of the large language model. We have three distinct modalities that need to be unified into the input space: compressed tokens, facial embeddings, and description embeddings. Therefore, three MLPs are trained alongside the autoregressive model.
We set the length of the context frames to 128, which results in the max sequence length around 5000. The global autoregressive model is trained for 3 days using 4 NVIDIA H100 GPUs with a constant learning rate of 2e-5 and 2k steps for warm-up.

To help the model distinguish the script for different frames, we inserted two special tokens, [START_OF_FRAME] and [END_OF_FRAME], at the beginning and end of each frame’s script, respectively. Furthermore, in our setting, there are three different modalities: face embedding, compressed tokens, and description embedding. The face embedding is extracted with FARL (Zheng et al., ), and for each frame, we extract the top 5 characters. The compressed tokens are generated by leveraging our trained encoder $\mathcal{E}$, with the target frame as input. The description is divided into different sentences, and each sentence is encoded into one [CLS] token using LongCLIP (Zhang et al., ). We treat the remaining text such as ”Characters:” and ”Scene Elements:” as identifiers, which are tokenized the same as LLaMA, as shown in Figure 2.

#### The identifier and descriptive texts in multimodal script.

Here, we provide an example to help readers understand the multimodal script in detail. Consider the example multimodal script in Figure 2, [*Characters:*], [*Scene Elements: *], [*Plots:*] and the symbol [*-*] are identifiers, while [*A man in a black suit…*], [*A woman with a teal headdress…*], [*Crowded event*], [*Elegent Lighting*], [*A man … a woman … are dancing closely…*] are text descriptions.
As shown in Figure 2, orange parts are identifiers, and blue parts are text descriptions. The identifier structures the input format, aiding the autoregressive model in understanding the different types of content within the script. The description is responsible for providing detailed information about the Character, Scene Elements, and Plot.

Figure: Figure 10: Additional Zero-Shot results.
Refer to caption: extracted/5912847/images/sup0/0.jpg

Figure: Figure 11: Additional Zero-Shot results.
Refer to caption: extracted/5912847/images/Gatsby/0.jpeg

Figure: Figure 12: Additional Zero-Shot results.
Refer to caption: extracted/5912847/images/the/0.jpeg

Figure: Figure 13: Additional Zero-Shot results.
Refer to caption: extracted/5912847/images/Good/0.jpeg

Figure: Figure 14: Additional Zero-Shot results.
Refer to caption: extracted/5912847/images/sup3/0.jpg

Figure: Figure 15: Additional story comparison results.
Refer to caption: extracted/5912847/images/additional_story_compare/ours/134.jpeg

#### ID-preserving Rendering.

To enhance our model’s ability to maintain character consistency, we additionally input the description embedding encoded by LongCLIP (Zhang et al., ) and the face embedding extracted by FARL (Zheng et al., ) into the trained decoder. Both the text embedding and face embedding are aligned with the decoder’s input using two-layer MLPs and then concatenated with the compressed tokens to serve as the input for cross-attention.

We assume a maximum of 15 sentences for description embedding and 5 character IDs. If the inputs do not meet these quantities, we pad them with zeros. Moreover, for the concatenated embedding, we randomly mask each token with a probability of 0.2. This encourages the model to learn the relationship between facial features, text descriptions, and the compressed token.

#### Continuous Token Modeling.

The $k$-mixutre GMM models the distribution $p(\mathbf{e}_{t}|\mathbf{H}_{<t},\mathbf{M}_{t})$, where $\mathbf{e}_{t}$ is the continuous real-valued image tokens of sequence length $S$ and dimension $d$. $\mathbf{H}_{<t}$ and $\mathbf{M}_{t}$ refer to the previous history information and the current multi-modal script. The GMM consists of $k$ multivariate Gaussian distribution components that are independent across the channel dimension $d$. To construct the GMM, we replace the logit prediction layer of our autoregressive model initialized using LLaMA with a linear layer with $2kd+k$ dimension for output. Subsequently, the mean $m$ and the variance $v$ of shape $S\times kd$ and mixture weights $w$ of shape $S\times k$ are derived. $m$ consists of a stack of tensors $[m^{(1)},m^{(2)},\dots m^{(k)}]$, where $m_{i}$ is of shape $S\times d$. Similarly, $v=[v^{(1)},v^{(2)},\dots v^{(k)}]$. The mixture weight is normalized along the channel dimension.
Given the ground truth image embedding $\mathbf{e}$, we aim to minimize its negative log-likelihood within the GMM:

$$ $\displaystyle-\log(p(\mathbf{e}|\mathbf{H}_{<t},\mathbf{M}_{t}))$ $\displaystyle=-\log\left(\prod_{l=1}^{S}\left(\sum_{i=1}^{k}w_{l}^{(i)}\prod_{ c=1}^{d}\mathcal{N}(e_{l,c}|m_{l,c}^{(i)},v_{l,c}^{(i)})\right)\right)$ (6) $\displaystyle=-\sum_{l=1}^{S}\log\left(\sum_{i=1}^{k}w_{l}^{(i)}\prod_{c=1}^{d }\mathcal{N}(e_{l,c}|m_{l,c}^{(i)},v_{l,c}^{(i)})\right),$ $$

During training, we additionally introduce the $\ell_{1}$ and $\ell_{2}$ loss between the predicted image tokens and the ground truth tokens to improve the model’s performance. However, we find that these losses are not on the same scale as the negative log-likelihood. The negative log-likelihood is way larger than the $\ell_{1}$ and $\ell_{2}$ loss, which reduces the influence of the $\ell_{1}$ and $\ell_{2}$ loss during the optimization.
The difference in scale is because the constant term and covariance matrix of the negative log-likelihood are directly related to the dimension $d$.
Concretely, for a variable $\mathbf{x}$ with mean $\mu$, the calculation of the negative log-likelihood is essentially represented as:

$$ $\operatorname{NLL}(\mathbf{x};\mu,\Sigma)=\frac{d}{2}\log(2\pi)+\frac{1}{2} \log|\Sigma|+\frac{1}{2}(\mathbf{x}-\mu)^{\top}\Sigma^{-1}(\mathbf{x}-\mu),$ (7) $$

where $\frac{d}{2}\log(2\pi)$ is the constant term and $\Sigma$ is the covariance matrix.
As $d$ increases, both the constant term and the size of the covariance matrix grow, resulting in a significant increase in the loss value. To bring the different losses to the same scale, we scale the negative log-likelihood loss by dividing it by the dimension $d$.
Consequently, the negative log-likelihood loss in Section 3.4 is formulated as:

$$ $\displaystyle L_{nll}$ $\displaystyle=\mathbb{E}(-\log(p(\mathbf{e}|\mathbf{H}_{<t},\mathbf{M}_{t})))$ (8) $\displaystyle=\mathbb{E}\left(-\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{L} \operatorname{log}p(e_{t,i}|e_{t,<i},\mathbf{e}_{<t},\mathbf{c}_{\leq t}, \mathbf{d}_{\leq t},\mathbf{i}_{\leq t})\right)$ $\displaystyle=\frac{1}{d}\cdot\mathbb{E}\left(-\log\left(\sum_{i=1}^{k}w_{l}^{ (i)}\prod_{c=1}^{d}\mathcal{N}(e_{l,c}|m_{l,c}^{(i)},v_{l,c}^{(i)})\right) \right).$ $$

## Appendix B Evaluation Metrics

We propose new metrics to evaluate short-term (ST) consistency and long-term (LT) consistency.
For short-term consistency, we select all the frames where a character appears. We then calculate the similarity of CLIP embedding of selected consecutive keyframes, as consecutive keyframes where the same character appears usually have high similarity.
For long-term consistency, We calculate the proportion of generated frames where the character appears to the ground truth frames where the character appears. For example, if the GT frames where a character appears are 1, 2, 3, 4, 5, and the generated frames where the character appears are 1, 2, 3, 6, then the long-term consistency is calculated as 3/5.
To select the frames where a character appears, we first select an image of the main character and use FARL to obtain the character’s embedding. We then compute the similarity of this embedding with FARL embeddings of all other frames and select images above a certain threshold.

The Aesthetic Score (Schuhmann et al., ) is calculated based on the CLIP image features. Each image is first passed through the CLIP image encoder to obtain an embedding. This embedding is then fed into an additional small neural network, which predicts an aesthetic score.

## Appendix C Multi-modal Scripts Data Pipeline Details

#### Multimodal script construction.

In this chapter, we explain in detail how we construct the multimodal script data.
Specifically,
We collect a massive movie dataset, with movies coming from Condensed Movies (Bain et al., ) and others collected using the same methodology as Condensed Movies. The data consists of around 23,000 long videos, which subsequently results in 5 million keyframes with corresponding script annotations. The overall multimodal script synthesis is as follows:

Figure: Figure 16: Multi-modal scripts data pipeline. We generate image captions using existing VLMs, and then further utilize LLMs to reformat the annotations into a structured script. We use the face retrieval method to extract different face embeddings of the same character, which benefits anti-overfitting data augmentation in Section 3.4.
Refer to caption: x4.png

- •
Extract keyframes from long videos at 1-second intervals, removing the beginning and end segments. These segments primarily contain sponsor messages, acknowledgments, and advertisements, lacking useful information. Subsequently, center-crop and resize each frame to a width of 896 pixels and a height of 512 pixels. Finally, leverage a Vision Language Model (Lu et al., ) to generate descriptions for the entire image.
- •
Using GroundingDINO (Ren et al., ) with ”Person” as the prompt, extract the top 5 characters from each frame. If the number of characters exceeds 5, the extra characters are ignored. If the number of characters is less than 5, there is no additional processing.
- •
Employ an existing Vision Language Model (Lu et al., ) to obtain captions for each segmented character. The face of each character is extracted to construct the face ID bank.
- •
Use a Large Language Model, taking the captions for the entire image extracted in step 1 and captions for each character in step 3 as input, to derive Scene Elements and Plot, and ask the model to output in a multimodal script format. At this point, we have structured Scene Elements and Plot but still lack Character. We then format all the character captions corresponding to each frame into the Character section of the multimodal script.
- •
However, the character face IDs for each frame are derived from the same frame, which can lead to the model overfitting by directly copy-pasting characters into the generated results. To address this, we use FARL (Zheng et al., ) to randomly retrieve the same character’s ID from the face ID bank as input. This strategy effectively mitigates the overfitting issue.

Figure: Figure 17: The overall data pipeline and corresponding prompts we use.
Refer to caption: x5.png

## Appendix D Additional Results

In this section, we present additional generated stories and videos. Our method exhibits strong generalization capabilities, producing high-quality results with inputs not included in the training set. Our method can understand text and character inputs, capable of generating results that align with the storyline while ensuring character identity consistency. The overall scene consistency is also well-preserved, as our autoregressive model is able to leverage context information as the reference. Most importantly, we are the first method to generate extremely long contents while preserving both short-term and long-term consistency. Besides, we even ensure that the IDs of different characters remain consistent before and after a camera change.

#### Generation process.

Our autoregressive model is similar to a multi-turn dialogue model. Specifically, we input the multimodal script corresponding to a keyframe, and the model autoregressively outputs two compressed tokens. Then, based on this, we input the multimodal script for the next frame to obtain the next two compressed tokens, without losing historical information. After obtaining all the compressed tokens, we input them along with the face embeddings and description embeddings from the multimodal script into the decoder to generate the keyframe. The keyframes are subsequently leveraged by image-to-video model to generate videos.

#### Additional results.

Our method effectively preserves both short-term and long-term consistency for multiple characters.
We present numerous results in Figure 10, 11, 12, 13 and  14. Our method is able to preserve multiple characters within long contents without the need for any fine-tuning. Even when the scene changes, our method effectively maintains the same character ID even during camera switches. Due to space limitations, we only select certain excerpts to showcase here. Please refer to the TITANIC generated by our MovieDreamer in supplementary material for longer results.

#### ID preservation.

We showcase our ability to preserve character ID in Figure 18. Our method exhibits strong generalization capabilities, allowing it to generate target characters without any further training.

Figure: Figure 18: ID Preservation results. Our method demonstrates the great ability to preserve ID. The compressed token for each keyframe is generated with the corresponding multimodal script as input, and the keyframes are rendered with compressed tokens, face embedding and description embedding as input. We show the target ID on the left.
Refer to caption: extracted/5912847/images/id_preserve/id_0.jpg

#### Video results.

The generated keyframes serve as the condition to generate videos. We show the results generated with our improved video generation model in Figure 19. Our keyframes have successfully preserved both short-term and long-term temporal consistency, facilitating the seamless rendering of these frames into a cohesive and harmonious video.

*Again,
we would like to highlight that our primary contribution lies in the hierarchical controllable very long generation of visual contents using MLLM for keyframe generation and diffusion model for video rendering.*
Any high-quality image-to-video model can be integrated with our approach. We showcase video results generated using higher-quality image-to-video models with our generated keyframes as input in Figure 20, which strongly demonstrates the effectiveness and flexibility of our method.
We believe our approach is highly promising has immense potential.

#### Different shot types and styles.

We find that although we did not explicitly model the camera shot types, the model can implicitly learn various shot types based on the multimodal script. The model is capable of selecting the most appropriate shot type based on the input. The shot types of anchor frames further assist the image-to-video model in generating videos with various shot types. Our method is also capable of generating cartoon results. We showcase the results in Figure 21.

Figure: Figure 19: Example video results with our video generation method.
Refer to caption: extracted/5912847/images/videos/14/0.png

Figure: Figure 20: Example video results using existing better image-to-video model LUMA. Existing image-to-video models can be integrated with our method to produce high-quality ultra-long videos, which demonstrates the flexibility and effectiveness of our MovieDreamer.
Refer to caption: x6.png

Figure: Figure 21: (a) Although our method does not explicitly model shot types, it can still implicitly learn various shot types and generate high-quality results with suitable shot types, demonstrating the potential of our approach. (b) Our method can also generate cartoon results and interleaved characters.
Refer to caption: x7.png

## Appendix E Additional Experiments

### E.1 Additional Comparisons

We showcase additional comparisons in Fig.15. Our method effectively preserves both short-term and long-term consistency, maintaining the overall style and character identity. Other methods exhibit increasing inconsistencies as the number of frames grows, such as changes in character appearance and shifts in style. Our method can generate extremely long content while nearly perfectly maintaining the consistency of multiple characters throughout.

### E.2 Additional Ablations on the Number of Compressed Tokens

#### Our goal and an important tradeoff.

While more tokens may yield better results, they sacrifice the context length of autoregression. Since we aim for long video generation, using as few tokens as possible is a reasonable design choice. Our goal is to achieve the highest quality images with a minimal number of tokens.

#### Why do we choose 2 tokens.

Existing work Paint-by-Example (Yang et al., ) demonstrates that a single token can effectively reconstruct the original image. Building on this discovery, we added a few additional tokens. We compared the reconstructed images using 8 tokens and 2 tokens, with the metrics shown in the accompanying table below. The metrics indicate that increasing the number of tokens does not offer significant improvements in results. As shown in Table 3, we calculate the inception score, aesthetic score as well as FID score of images of different numbers of tokens. We further compare the CLIP similarity between reconstructed images and target images. The evaluation metrics demonstrate that the results of 8 tokens are slightly improved. But considering our objective of long content generation, this slight improvement is not enough for us to prioritize it when making trade-offs regarding sequence length and our limited computational resources, as 8 tokens will lead to a much shorter sequence length and demand more computational overhead. Consequently, we eventually chose 2 tokens. But we believe that if there are enough computational resources, 8 tokens will be a better choice.

**Table 3: Quantitative ablation study on the number of compressed tokens.**
| Method | CLIP-sim$\uparrow$ | Inception$\uparrow$ | Aesthetic$\uparrow$ | FID$\downarrow$ |
| --- | --- | --- | --- | --- |
| 2 tokens | 0.825 | 8.235 | 6.378 | 0.406 |
| 8 tokens | 0.841 | 8.193 | 6.457 | 0.348 |

## Appendix F Limitations and Future Works

Figure: Figure 22: When there are too many characters in one frame, the model sometimes confuses the character IDs.
Refer to caption: x8.png

Despite our method demonstrating strong potential in generating long sequences, it still has some limitations. (1) We employ LongCLIP to compress each sentence of descriptive text into a single token. While this strategy effectively reduces the sequence length, it provides only a coarse representation of each sentence, which causes information loss.
(2) Our method’s ability to preserve object consistency is limited. We did not construct the multi-modal script with scene information, as the costs far exceed our capacity. Moreover, as shown in Fig. 22, if there are too many people in one frame, our method is prone to mixing the appearance of characters.
(3) Analogous to other large models, our approach requires substantial data and computational power for training.
(4) Currently, we generate 128 keyframes at a time and run multiple iterations of this process to create long content. However, our goal is to extend the maximum token length, fundamentally addressing the long sequence generation by using long sequence generation methods in LLMs. Due to resource constraints, we are unable to pursue this approach at present and leave it in future work.

#### Social impact.

Our method can generate high-quality long stories and videos, significantly lowering the barrier for individuals to create desired visually appealing content.
However, it is essential to address the negative potential social impact of our method. Malicious users may exploit its capabilities to generate inappropriate or harmful content. Consequently, it is imperative to emphasize the responsible and ethical utilization of our method, under the collective supervision and governance of society as a whole. Ensuring appropriate usage practices requires a collaborative effort from various people, including researchers, policymakers, and the wider community.
Furthermore, the code, model, as well as the data, will be fully released to improve the development of related fields.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Since authors are changeing LLaMA input to CLIP embedding (for sentence), does it cause much for the LLM to adapt to such input?
How much cherry-pick do you need for a single long video?
soundness: 4
strengths: The quality is GREAT!
Engineering wise, this paper produces a product-level solution to long-video (movie) generation; Scientificly, this paper also explores training a MLLM to output visual features given the input scripts, instead of simply using an agent framework to compose scripts, and use subject-driven models to preserve character identity.
The ID Preserving is also very effective
summary: This paper presents MovieDreamer to make long video. This is challenging because current video diffusion models can only deal with short clips and do not have character consistency. The model trains a MLLM that can predicts the visual tokens with movie script conditioning. Then using the diffusion decoder to decode the condition into key frames, and using I2V to render it into video clips.
weaknesses: Lack of non-human results, don't know if such methods can generalize well to general domain long-vedio generation.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please provide the evaluation of the ID consistency of the model, clarify the technical contributions, and discuss the limitations of the framework.
soundness: 3
strengths: + The model provides an effective integration of autoregressive model and diffusion models, for the long visual sequence generation task. The combination is reasonable by successfully leveraging the arbitrary-length generation capability or AR model and the powerful rendering ability of diffusion models. 
+ The utilization of the gaussian mixture model for the continuous visual token prediction is reasonable and inspiring.
summary: This paper presents a hierarchical generation framework for long video sequence generation, which can be potentially applied to movie video generation. The key idea is to leverage the autoregressive models to generate key frame tokens, and then use diffusion models to decode the tokens into RGB frames. This approach leverages the advantage of any-length future prediction ability of AR model and the powerful rendering ability of diffusion models. Through some other techniques such as multimodal script, face embedding condition, etc., the framework can generate ultra-long video sequences with better quality.
weaknesses: - The paper proposes a lot of techniques to improve the identity consistency among different clips, however, I cannot find any ID-related evaluation on the generation results. 
- The paper did not analyze the boundaries of the proposed method. For example, 
    - when will the model fail at ID consistency since the model only provides one face embedding as the condition. 
- The technical novelty is relatively limited. There exist many multimodal generative models can do the similar key frame generation task, though they may not be directly used for movie key generation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I have outlined some concerns and suggestions in the weaknesses section, and there are additional questions that need clarification.

1. Is the covariance matrix in Eq. 7 of the supplementary material a diagonal matrix?

2. In the few-shot training section, what is the “episode’s visual tokens” (line 302)?

3. In Fig 8, what are the reference images for generating the few-shot results?  If I understand correctly, face embedding is also used as part of the multimodal script to generate zero-shot results. For the few-shot generation, what images are used as references, and how do these reference images help improve identity consistency?
soundness: 3
strengths: 1.	The proposed framework seems promising for generating long video. 
2.	The proposed method can effectively generate character-consistent story keyframes, and the experimental results show the superiority of the proposed method to other state-of-the-art methods.
summary: Existing diffusion model-based video generation models can only support short video generation and lack understanding of narrative structures and plot progressions. Therefore, this paper explores long-story video generation by leveraging the inference capability of autoregressive generation and video diffusion models’ rendering capability. Specifically, this paper proposes a hierarchical framework and a structured multimodal story input representation. The framework fine-tunes the multimodal large model LLAVA to generate compressed tokens of keyframes and fine-tunes SDXL as a decoder to generate keyframe images from the compressed tokens. Finally, SVD is used to generate the video. Additionally, the paper proposes a few-shot training method, enabling the model to achieve customized generation. Extensive experiments compare the proposed method with existing story keyframe generation methods, demonstrating superior performance.
weaknesses: The overall writing of the paper is relatively clear, and the experimental results are promising. However, some of the descriptions appear to be incorrect or misleading, and the experiments are also not entirely comprehensive.

1. In line 242, why are the parameters of the GMM stated as 2kd means and 2kd variances? According to the reference work, each GMM parameter should contain kd means, kd variances, and k coefficients for each compressed token.

2. What would the results look like if Eq. 2 were used as the sole objective function? There doesn't appear to be a corresponding analysis of this in the experiments. Is the inclusion of L2 and l2 necessary, or have previous works discussed the necessity of incorporating these terms?

3. In Fig 2, if I understand correctly, the multimodal script in the top-left corner is used to generate a single keyframe. However, the three keyframes shown on the right under the VLM section could easily give the impression that they were all generated by the same script. This may lead to some confusion for readers. Additionally, according to Equation 1, the input to the decoder should also include a noise latent. However, the inference process shown at the bottom of Figure 2 does not fully display the complete inputs and outputs, which could lead to some misunderstandings.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. The proposed framework propose a hierarchical way to combine autoregressive modeling and diffusion image generator together for long-term complex video generation.
2. The paper proposed several novel techniques, e.g., continuous token supervision, anti-overfitting strategies, id-preserving, to improve the performance of the framework.
3. The proposed method achieved state-of-the-art performance on quantitative evaluation.
summary: The paper proposed a novel hierarchical video generation framework MovieDreamer, which is able to generate long-duration videos with high visual fidelity. The framework consists of a SDXL-based visual autoencoder and a multimodality VLM. The autoencoder focuses on obtaining the visual tokens of input images, as well as rendering targeted keyframes, while VLM autoregressively predicts visual tokens. Quantitative evaluation shows that the proposed method outperforms sota in long video generation.
weaknesses: 1. L212: Eq1, the DAE has not been clearly defined in the paper. Also, in Fig2, I could not find DAE in the pipeline. I hope the authors could clarify this.
2. It is a bit confusing whether Eq1 is the real training loss or not? According to Sec3.3 and Fig2, the input of SDXL should be image token, face embedding and description embedding, which does not align with Eq1. It is better to clarify the final training loss.
3. It is not clear how random masking strategies work during training and inference stages.
4. The high dropout rate (50%) seems to be unusual. Are all the parameters updated in VLM? Have the authors tried only tuning part of the network?
5. L323: if the feature of the first frame is always used, will generated character be limited to small motion, and limited content? For example, for large rotation and movement, the performance of the proposed method may drop?
6. What are the frame numbers of full-length video?
7. From Fig8, while compared to zero-shot setting, the few-shot setting indeed improves the ID-preserving ability. However, there is still an obvious gap between target ID and generated image. What could be the potential reason for this? Any solution?
8. What is the training cost? And what are the inference time and GPU memory requirements?  
9. For quantitative evaluation, it seems that the reported metrics all focus on image level. They can not evaluate how good the performance of spatio-temporal modeling. It is better the authors could evaluate the proposed method using other metrics for video generation.

Meta-review: The paper proposes a novel long-duration video generation framework, which is based on fine-tuning LLAVA to generate compressed tokens of key frames, which are then used to generate keyframes using a fine-tuned SDXL, and then videos using SVD. Extensive experiments demonstrate the effectiveness.

Initial reviews had mixed ratings 5686. Reviewers raised a number of concerns, including:
1. lack results of non-human videos (zniU)
2. any cherry picking done? (zniU)
3. missing evaluation on ID (xVaX)
4. boundary cases (limitations) not analyzed, e.g., will the model fail at ID consistency if one face embedding is provided (xVaX)
5. limited novelty (existing methods can do key-frame generation) (xVaX)
6. missing or unclear descriptions about method (4ibH, Jjeo) 
7. missing ablation study on objective function (4ibH)
8. training cost? (Jjeo)
9. still obvious gap for target ID evaluation (Jjeo).
10. need to use video generation metrics, not just image-level (Jjeo). 

The authors provided a response to address these concerns, including providing ablation studies, discussing limitations, and expanding on the novelty.  During discussion, authors clarified many points with 4ibH and added more interpretations of the various results / ablations. In the end, all reviewers were satisfied after the rebuttal and discussion, and final reviews were all positive 6688.  Reviewers appreciated the quality of the result, as well as the unique problem addressed.

Authors should update the paper based on the reviews, response, and discussions.

Decision: Accept (Poster)