MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences
Authors: Canyu Zhao
Mingyu Liu
Wen Wang
Weihua Chen
Fan Wang
Hao Chen
Bo Zhang
Chunhua Shen
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Model
Vison Language Model
Long Video Generation
Long Story Generation
Primary area: generative models

Abstract: Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities.

Published as a conference paper at ICLR 2025
MOVIEDREAMER: HIERARCHICAL GENERATION FOR COHERENT LONG VISUAL SEQUENCES
Canyu Zhao Mingyu Liu
Wen Wang'
Weihua Chen"
Fan Wang"
Hao Chen' Bo Zhang' Chunhua Shen'
'Zhejiang University
2 Aliiaba Group
ABSTRACT
Recent advancements in video generation have primarily leveraged diffusion mod- els for short-duration content. However, these approaches often fall short in mod- eling complex narratives and maintaining character consistency over extended pe- riods, which essential for long-form video production like movies. We pro- pose MrvieDreammrr a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our ap- proach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene de- scriptions with detailed character information and visual style, enhancing continu- ity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities.
INTRODUCTION
Driven by the advent of generative modeling techniques (Rombach et al., 2022 Podell et al. 20233
there has been significant progress in video generation in the research community (Guo al. 2024 Ho et al. 2022b; Blattmann et al. 2023). Tremendous research efforts have focused adapting pre-trained text-to-image diffusion model (Guo et al., 2024 Ho et al. 2022b) to a video generation model. Recently revolutionary leap has been made with the Sora (Brooks et al. 2024) model which transformer model. This model demonstrates remarkable substantially scales a spatial-tempont extending the duration of generated videos from seconds to video generation quality, dramatically an unprecedented full minute. This technology profoundly reshapes the boundaries of generative AI, fostering anticipation towards hours-long movie production.
While predominant video generation methods adopt diffusion models, this paradigm is more suited for visual rendering and is less adept at modeling complex abstract logic and reasoning compared to autoregressive models, as evidenced in natural language processing. Moreover, diffusion models lack the flexibility support arbitrary length. On the other hand, autoregressive models (Wang al.
2023a Jiang et al, 2022. Hu et al. 2023 Pan et al., 2023) have shown superior ability in handling complex reasoning and are better at predicting what might happen in the next-a a key element of a world model. Moreover, autoregressive models offer greater flexibility in handling varied lengths of predictions and can also benefit from mature training and inference infrastructure.
such, there are attempts that use the autoregressive model for video generation (Kondratyuk al. [023] Hu al. 2023). However, autoregressive modeling is not as compute efficient as diffusion model for visual rendering and demands substantially more computing
for image rendering.
Recent progress (Henschel et al. 2024 Chen et al. 2023 Qiu et al. 2023 Brooks et al. 2024) sug- gests that video generation quality in short duration (from seconds minute) can be consistently improved by scaling the compute due to the scaling law. In this work, we sidestep the short video generation problem and instead seek generative framework for long video generation featuring

---

Published as a conference paper at ICLR 2025
Figure 1:
The Titanic movie scenes generated by the proposed MovamDr The generation is conditioned on the elaborate multimodal movie script. The original movie is unseen during training. Please refer to Figure 10
13 14 for more results.
complex narrative structures and intricate plot progressions, which is hardly achieved by merely scaling the current approaches. Our aim is push the boundaries of video generation beyond the limitations of short-duration content, enabling the creation of videos with rich, engaging storylines.
In this paper, we introduce a hierarchical approach, dubbed MovieDreamer, that marries autoregres- sive modeling with diffusion- based rendering to achieve novel synthesis of long-term coherence and short-term fidelity in visual storytelling. Our method leverages the strengths of autoregressive models to ensure global consistency in key movie elements like character identities and cinematic styles, even when the camera switches back and forth. The model predicts visual tokens for spe- cific local timespans, which are then decoded into keyframes and dynamically rendered into video sequences using diffusion rendering. Our method mirrors the traditional movie directing process, where intricate plots are decomposed into distinct, manageable scenes. Our method ensures that both the overarching narrative and the minute spatial-tempon details are maintained with high fidelity, thus enabling the automated production of movies that are both visually cohesive and contextually rich.
To this end, represent the whole movie using sparse keyframes and employ diffusion autoen- coder to tokenize each keyframe into compact visual tokens. We train an autoregressive model to predict the sequence of visual tokens. This model is initialized from pretrained large language mode, a 7B LLaMA model (Touvron et al. 2023). to leverage its rich world knowledge for better generalizability. As such, the model can be regarded as multimodal model, which, rather than outputting the text, predicts the visual tokens with movie script conditioning.
Three key designs need be specially considered during the autoregressive training. One to resort to various model and input perturbation techniques to combat the overfitting tendency since the high-quality movie training data limited. Besides, we propose to condition the model using novel multimodal script, which, for each keyframe, includes the rich description of the image style and scene elements, along with detailed text description of characters well as their retrieved face embeddings. Such multimodal script facilitates narrative continuity across different segments of the movie while offering flcxibility for character control. Moreover, we stochastically append a few reference frames of the same episode the beginning of the input, and in this way, the model acquires in-context learning capability and can synthesize higher-quality personalized results. Such feature is particularly useful in generating coherent continuations of given movie episodes.
We subsequently decode the predicted vision tokens using an image diffusion decoder and further render the image the video clip using image-to-video diffusion model. Importantly, enhance
2

---

Published as a conference paper at ICLR 2025
this process by finetuning an edennity-presenvine diffusion decoder. This refinement effectively im- proves identity preservation in the resultant video clips. Our work is essentially orthogonal to the current efforts aimed at improving the short video generation quality via compute scaling and our method can benefit from these progresses. Our approach is validated through extensive experiments on wide range of movie genres, showeasing excelling ability to generate visually stunning and coherent long-form videos over the state-of-the-art.
To summarize, the contributions of this paper are:
We introduce MovieDreamer, novel hierarchical framework that marries autoregressive models with diffusion rendering to generate long-term coherent stories and videos with multiple charac- ters well-preserved. The method substantially extends the duration of generated video content to thousands keyframes. To the best of our knowledge, we are the first to tackle the ultra-long narrative video generation problem, rather than merely clip looping.
We generate the visual token sequence using a multimodal autoregressive model. The autore- gressive model supports zero-shot and few-shot personalized generation scenarios and supports variable length of keyframe prediction.
We use a novel multimodal script to hierarchically structure rich descriptions of scenes, plots, well as the character's 's identity. This approach not only facilitates narrative continuity across different segments of a video but also enhances character control and identity preservation.
Our method demonstrates superior generation quality with detailed visual continuity, high-fidelity visual details, and the character's 's adentity-preservinng ability. Both story generation and video generation can be addressed with our method.
RELATED WORK
Video generation models. Video generation has advanced mainly through diffusion and autore- gression methods. The popularization of Stable Diffusion (Rombach et al.
(202))
has sparked exten-
2024: Ho al. 2022b a He et al. 2023). For long videos, strategies like (Yin et al. 2023][Qin et al.
sive studies, improving video quality with various techniques (Guo al.
2023 Wang et al. 2023b
Chen et al. 2023 Henschel al. 2024) can only generate simple results. Despite their high com- putational load and challenges in managing complex sequences, diffusion models are widely used. Autoregression, on the other hand, is less common but excels in specific areas like autonomous driving by employing unified token spaces (Hu et al. 2023), although it struggles with continuity in complex scenes. Our approach combines the strengths both methods in hierarchical and scalable manner, enhancing control and flexibility irrespective of video length.
Multimodal large language models. Recent advancements in NLP lead to the emergence of vi-
sual language models (VLMs) through enabling large language model (Touvron et al., 2023) to
comprehend visual modalities (Liu al. 2023b Bai al. 2023a b Chen et al. 2024). In addition
to language, VLMs have demonstrated their power in generating images (Pan et al. 2023] 3 Sun et al.)
2023c b). Moreover, ongoing research (Zhang et al.
2023 Lin et al.) 2023 Li al. 2023a| Zhao
et al. 2022) has showcased the robust capabilities of Multimodal Language Models (MLLMs) in
comprehending video content. Some works (Cai et al.
2024 Cheng al. 2024 Ren al. 2024b
Rasheed et al. 2024; Lai et al. 2024) enable VLMs to assist in spatial und understanding and other visual tasks. VLMs (Jiang et al., 2022 Kim et al. 2024 Etukuru al. 2024 Brohan et al. 2023 Driess et al.) 2023) also demonstrate the powerful capabilities in robotics. These advancements have solidified the effectiveness of VLMs addressing complex problems. Building on this foundation, our approach leverages VLMs to facilitate the generation of extremely long video content.
Visual story generation. The crux of visual story generation lies in ensuring consistency across generated images. Similar to video generation, diffusion and autoregression are also two primary manners. Existing diffusion-based methods predominantly employ conditional generation to main- tain the coherence (Su et al.) 2023 Wang et al., 2023c Tewel et al.) 2024 Hertz et al.] 2023) Some (Li et al. 2023b Wu et al. 2024) have shown promising results in retaining high-quality fa-
cial features and identity. Other approaches are based on the autoregressive ideas (Feng et al, 2023
Liu et al. 2023a Rahman et al. 2023 Pan al. 2022). However, these approaches generally yield
3

---

Published as conference paper at ICLR 2025
Diffusion Autoencoder (DAE)
Inference
Compressor
Inbodding
SDXL
Veaal
Roby
Encoder
Decoder
Enebadding
Multimodal Soripts Example
Auto-regressive Prediction
Chanscters:
mon black suit with slicked-Be sredd-back hain. FFose TD= womon with a teal henddress and matching mackiace Fooe Tbo
Multimodal Scripts 3225
Multimodal Scripts 3226
Scere Elemants:
Croweed event
Multimodal Modeling with VLM
Elegant lighting..
Plat:
mon women are dancing closely of o crowded event
Thene is subtle but polpable sonse connection confrontation betweon mon women, suggesteed by each gaze and prexiiiiy setting
crowded
Keyframe
videa Gen
KeyFrame 3225
Image- ti-ee
KeyFrame
Figure 2:
The framework of our MovieDreamer. Our autoregressive model takes multimodal scripts as imput and predicts the tokens for keyframes. These tokens are then rendered into images, forming anchor frames for extended video generation. Our method ensures long-term coherence and short- term fidelity in visual storytelling with the character's identity well preserved.
mediocre results and are challenging to scale up. Story and vidco generation share intrinsic links, with overlapping solutions. Our method uniquely integrates these, offering unified approach.
METHOD
3.1 OVERVIEW
We propose a novel framework for generating extended video sequences that leverages the strengths of autoregressive models for long-term temporal consistency and diffusion models for high-quality image and video rendering. Our method takes multimodal scripts as conditions, predicts key frame tokens in an autoregressive manner, renders the tokens into images and uses these images as anchors to produce full-length videos. Our method offers flexibility to support zero-shot generation along with few-shot scenarios which the generation results are required to follow the given reference. We take special care of preserving the identity of the characters throughout the multimodal script design, autoregressive training, and diffusion rendering. illustrate the overall framework in Figure
3.2 KEYFRAME TOKENIZATION VIA DIFFUSION AUTOENCODER
To create a concise yet faithful representation of images, we employ diffusion autoencoder. Our encoder, contains a pretrained CLIP vision model (Radford al. 2021 and transformer-based token compressor, which encodes an image x into reduced number of compressed tokens denoted by c. The decoder, structed image by leveraging the compressed tokens in the cross-attention module. We train the compressor and the decoder while leaving the CLIP vision model frozen. The loss for training this diffusion autoencoder is follows:
finetuned based on the pretrained SDXL and yields an 896x512 recon-
LDAE = Ex (xx.e)) ||e D(z,, (xx)))
(1)
where Xn is the input image and z, = a,x + O,E is its corresponding noisy latent at timestep t. (/O 7) is the Gaussian noise. Or and or define the noise schedule. In our experiment,
4

---

Published as a conference paper at ICLR 2025
empirically find that merely two tokens can sufficiently characterize major semantics of keyframes, corroborating the finding of previous work (Yang et al. 2002).
3.3 PRESEEERNN DIFFUSION RENDERING
In our approach, while the primary diffusion decoder D adeptly reconstructs target images, it occa- sionally falls short in capturing fine-grained details, notably in facial features, due to the attenuation of details in compressed tokens. To address this, we enhance the cross- -ross-attention module within D. This enhancement involves the integration of both descriptive text embedding, d and face em- bedding, f, derived from the multimodal script. Specifically, the face embedding f and description embedding d is projected to the same dimension as the compressed token with two additional MLPs, which are then concatenated together and serve as the input to the keys and values of cross-attention modules in D. The training loss is:
DAA = Wo (i
f, d)l12
(2)
To further advocate the model's s ability to focus on critical details, we introduce a random masking strategy that obscures subset of the input tokens as zero tokens only during training. This technique encourages the decoder to more effectively utilize the available facial and textual cues to reconstruct the images with higher fidelity, particularly in preserving "dentity-specifie characteristics. This ID- preserving rendering also compensates for the loss of identity during the autoregressive modeling, leading to substantially improved identity perception quality illustrated in Figure 3
3.4 AUTOREGRESSIVE KEYFRAME TOKEN GENERATION
We initiate our approach by leveraging pretrained large language model, specifically LLaMA2- 7B (Touvron et al. 2023), construct our autoregressive model, g. Unlike traditional LLMs, g is designed to predict compressed vision tokens from the multimodal script M of the corresponding keyframe and historical information H, which is formulated as e; = ((H,,M, M,).
Traditional LLMs often employ cross-entropy loss for training, which is suitable for discrete out- puts. However, our model deals with continuous real-valued image tokens, which makes cross- entropy inapplicable. Inspired by Tschannen et al. (2024), we adopt k-mixture Gaussian Mixture Model (GMM) to effectively model the distribution of these real-valued tokens. This involves pa- rameterizing the GMM with kd means, kd variances, and & mixing coefficients. These parameters are predicted by attaching an additional trainable linear layer at the end of the autoregressive model, enabling the construction of the GMM and the sampling of continuous tokens from the GMM. The model is trained by minimizing the negative log-likelihood:
T
Lall = B(-
log
(3)
1=1=1
where T represents the total number of frames, t indexes the current frame, and L is the number of compressed tokens per frame. To ease the learning for these continuous tokens, we also incorporate & and lz losses between the predicted and ground truth token and optimize the following objective:
for Call (Epred, egt) f2(epred: egt).
Anti-overfitting It is challenging to massively acquire large amounts of high-quality, long- duration video data. We managed to collect 50k long videos, yielding 5M keyframes for autore- gressive training. Yet this data volume is relatively limited, leading to severe overfitting issue where the generative model fails to generalize well during testing. To address this, we implement several strategies which are crucial for training:
Data augmentation. To maximally utilize our training data, we apply random horizontal flips and stochastically reverse the temporal order of video frames. Such training data augmentation considerably increases the diversity of training data.
Face embedding randomization. To prevent identity leakage randomly retrieve face embed- dings of the same character from different frames. Otherwise, the model will memorize the train- ing frame simply from the face embedding input.
5

---

Published as a conference paper at ICLR 2025
Aggressive dropout. An unusually high dropout rate of 50% is utilized, which is crucial for generalized learning from limited training data.
Token masking. We incorporate random masking of input tokens with a probability 0.015, which is applied to the causal attention mask. This forces the model to infer missing information based on the available context (face ID), further strengthening its ability to generalize from partial data.
Multi-modal scripts for autoregressive conditioning. We develop well-structured multi-moda script format to serve as input for autoregressive models, as depicted in Figure [16] Our script inte- grates multiple dimensions: characters, scene elements, and plots. Accurately representing character appearance using text alone is challenging; therefore, we combine textual descriptions with facial embeddings (Zheng al., 2022) to provide a more detailed representation of each character. To facilitate the processing by the autoregressive model, we structure the script format to distinctly delineate these elements.
For non-textual modalities such as facial embeddings and compressed tokens are projected to LLaMA's embedding space using multi-layer perceptron. The primary challenge arises with tex- tual data, which tends to produce long sequences, thereby consuming excessive token space and limiting the model's contextual breadth. This challenge poses significant obstacle to achieving goal of long content generation. To mitigate this, treat text separate modality, dividing it into "identifiers" and "descriptions" (claborated in Appendix [A]. Identifiers establish the script's structure, while descriptions detail the attributes for generation. Descriptions sentences, with each encoded into a single [CLS] token using CLIP and then projected into the unified input space. In other words, perform the sentence-level tokenization.
segmented into
This method significantly extends the usable contextual length during training by condensing an text encoder for descriptions, supporting inputs up to 248 tokens, which enhances our capacity to handle de- tailed narrative content. Consequently, the multi-modal script at timestep & and preceding historical information are represented as:
entire sentence into single token. We utilize LongCLIP (Zhang et al.) 2024)
(5) where d, i, and e₂ denote the embeddings for descriptions, identifiers, and characters' facial em- beddings, respectively. Cet represents the previously predicted compressed frame tokens. The negative log- likelihoodd loss in Equation 3
M. = [d., ie, c.], H = (de,, Cer, ecr),
is then formulated as:
T
Lnll = E(-
logpler.ee. cen dsn.is)))
(6)
Few-shot training for personalized generation. promote personalized movie content gener- ation, propose a few-shot learning method that utilizes in-context learning. During training, select random frames from an episode, encode them into visual tokens, and prepend these to- kens stochastically to the input. This strategy not only promotes in-context learning, allowing the model to tailor content based on the reference frames, but also acts as data augmentation technique, effectively mitigating overfitting.
Our model is versatile, supporting both zero-shot and few-shot generation modes. In zero-shot mode, the model generates content from the multimodal script. In few-shot mode, it leverages small set of user-provided reference images to align the generated content more closely with user preferences, without necessitating further training. This capability ensures that users can efficiently produce high-quality, customized visual content that aligns better with their desired target.
3.5 KEYFRAME-BANE BASED VIDEO GENERATION
After obtaining the keyframes, we can generate video clips of the movie based on these keyframes. A straightforward approach is to utilize existing image-to-video model, such as Stable Video Diffusion (SVD) Blattmann et al. 2023 to generate these clips. Specifically. SVD transforms the input image into latent features for conditioning and introduces interaction with the CLIP features of the input image via cross-attention. Although SVD is capable of generating high-quality short videos, e.g., 25 frames, it struggles to generate longer movie clips.

---

Published as a conference paper at ICLR 2025
Target
Before
After
Figure 3: The ID-preserving renderer significantly enhances the perceived identity of the character. For the second row, the imput is only the compressed token. For the third row, the input is the compressed token, face ID, and description embedding.
To generate longer movie clips, straightforward way is to utilize the last frame of the previously generated video the initial frame for generating the subsequent video. This process can be iter- ated to obtain a lengthy video sequence. However, we empirically found that this causes serious accumulation of errors: the quality of the video frames gradually deteriorates the time gets longer.
To tackle this, we propose a simple and effective solution. Our motivation is to always use the feature of the first frame an "anchor" during video extension, to enhance the model's awareness of the original image distribution. In practice, we use the CLIP feature of the original input image, instead of the last frame of the previous video, for cross-attn interaction when generating subsequent videos.
We want emphasize that key contribution lies in the hierarchical controllable very long video generation, with MLLM for keyfrume generation and diffusson model for video rendering. Our subsequent experiments also demonstrate that existing high-quality video generation models trained by resource-rich institutions can be integrated with our MovieDrcamer, producing better visually appealing results. This further demonstrates our flexibility.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We use pretrained models to reduce the training computational overhead, which is very common in the era of large models. We use the U-Net of pretrained SDXL (Podell et al., (023) as our diffusion decoder. The encoder consists of a pretrained CLIP image encoder (ViT-L) (Radford et al.) 2021) and a transformer-
compressor. The autoregressive model initialized with a pretrained LLaMA-7B (Touvron et al. 2023) checkpoint. All MLPs used to align different modal inputs follow the architecture in LLa VA (Liu et al. 2023b). Please refer to Appendix for more details about the network architecture and training.
4.2 EXPERIMENTAL SETUP
Dataset. The majority of our training data is sourced from Condensed Movies (Bain et al. 2020 with additional data collected using the same methodology as this dataset, resulting in 5M keyframes with corresponding script annotations. To systematically evaluate the effectiveness of our method, we construct test dataset consisting 100 long movies that are NOT included in the training set, with IM keyframes after pre-processing. Each of these videos is processed through our proposed data pre-processing pipeline (see Appendix C which accurately extracts and annotates keyframes.
Evaluation metrics. We evaluate the effectiveness of various methods using the following metries. The CLIP score (Radford et al.) (02II assesses semantic alignment between the plot and generated outputs. Quality is evaluated using the Aesthetic Score (AS) (Schuhmann al. 2022). Fréchet

---

Published as a conference paper at ICLR 2025
Image Distance (FID) (Heusel al. 2018), and Inception Score (IS) (Salimans et al. 2016 For assessing consistency, we propose new metrics for short-term (ST) and long-term (LT) consistency. ST is measured by calculating the CLIP cosine similarity between adjacent frames containing the target character. ITT consistency is evaluated by checking the presence of the target character in selected frames across the test Finally, calculate the ratio of frames containing the target character to the total number of selected frames as measure of the accuracy in maintaining long- character consistency. Please refer the Appendix B for more details about the evaluation metrics.
wearing suit
people in the backeround
gaze al camera
sitting, serious looking
Ours
StoryDiffusion
Story Gen
Figure 4: Qualitative comparisons of story generation. Please refer to the appendix for more comparisons. Each keyframe is generated with the corresponding multimodal script as input.
4.3 COMPARISON WITH THE STATE-OF-THE-ART
Story generation. Many existing story generation methods focus on fine-tuning with small datasets, exhibiting poor generalization. Consequently, compare our method only with those that demonstrate high generalization capabilities, namely StoryDiffusion (Zhou et al.) 2024) and StoryGen (Liu al. 2023a). As shown in Figure StoryDiffusion consumes significant compu- tational overhead and tends to generate generic faces that do not align with the desired character. Similarly, StoryGen fails to preserve consistency and generates abnormal results. contrast, our method achieves generating extremely long content while preserving both short-term and long-term consistency across multiple characters. The observation also evidenced by the quantitative re- sults in Table [ where method achieves high scores in both LT and ST. Moreover, the higher CLIP score reflects that our generated results align well with the storyline. Better IS, AS, and FID scores demonstrate our method achieves high-quality images. We showcase more qualitative results in Appendix E
KeyFrame25
KeyFrame32
KeyFrame37
KeyFrame43
W Face
w/o Face ID
Figure 5: Ablations on face embedding. Character consistency is well- preserved with face embed- ding. The autoregressive model generates compressed tokens with multimodal script input. The results are rendered by the decoder D, with compressed tokens as well as face embedding and de- scription embedding as input.
8

---

Published as a conference paper at ICLR 2025
Table Quantitative comparisons of different methods. ST and LT refer to our proposed short-term and long-term consistency metrics, respectively. noC denotes without continuous token supervision.
Method
CLIP
Inception
Aesthetic
FID
ST
LT
StoryDiffusion
15.232
8.739
6.134
4.643
0.627
0.596
StoryGen
13.261
6.216
3,837
8.557
0.508
0.542
Ours
19.584
9.698
6.093
2.043
0.646
0,814
Ours-ref
20.071
9.842
6.288
1.912
0.701
0.893
Ours-noC
18,833
8.609
5,893
2.714
0.606
0.755
Ours-noC-ref
19.431
8.774
5.979
2.560
0.616
0.776
Table 2: Quantitative comparison full-length video generation.
Method
CLIP
Inceptiont
Aesthetict
CLIP-simt
StreamingT2V
15.304
7.636
4.119
0.624
SEINE
19.404
7.463
5.825
0.688
FreeNoise
14.327
6.532
3.194
0.612
Ours
19.520
8.642
6.049
0.704
Video results. We perform a detailed comparison between our method and existing methods for generating long videos. For text-to-video methods, test set as input. For image-to-video approaches, we employ keyframes generated by our methods inputs. As illustrated in Table 2 and Figure [6] (b), our method significantly outperforms prior open-source models terms of quality, demonstrating strong generalization capabilities. Most im- portantly, our method is capable of generating videos that last tens of minutes or even hours with little compromise in quality, achieving state-of-the-ar results. Here, we compare existing methods with our enhanced video model. As shown in Appendix D when integrated with enterprise-level closed-source video models, our approach demonstrates even better results.
detailed descriptions prepared in the
4.4 ANALYSIS
Continuous token supervision. We noticed a significant decline the model's ability to maintain ID consistency the test set when using only LI and L.2 supervision. We believe this is because L1 and L22 losses are not good at modeling probability distributions, as they are more suitable for need to model the complex conditional distribution of the autoregres-
regression tasks. However, sive process. Therefore, we applied more refined supervision on continuous real-valued tokens. As illustrated Figure with continuous token supervision, the model better capable of preserving target ID. Both the quality of the generated image and the preservation of character ID degenerate without continuous token supervision.
Ours
SEINE
StreamingT2V
FreeNoise
Con
eifo Con
(H)
(b)
Figure 6: (a) With continuous token supervision, our model demonstrates stronger generalization capabilities and achieves higher-quality results. Better character IDD is preserved with continuous token supervision. (b) Visualization of long video generated by different methods.
Anti-overfitting strategies. Large autoregressive models are powerful learners, making casy for them to overfit the dataset. As shown in the first in Figure 7 the generated contents dominated by the input character. The model produces similar visual content even when given different text prompts. Our anti-overfitting strategies are designed to weaken the correspondence between character IDs and target frames, thus avoiding simple memorization. As can be seen in the second row, helps produce diverse high-quality results that align well with the description.

---

Published as a conference paper at ICLR 2025
Before
After
target ID
wearing red shirt
close up of face, upset
talking
raisc his arms
Figure 7: After resolving the overfitting, our model is capable of generating more diverse results.
Each keyframe generated with the corresponding multimodal script as input.
Few-shot Zero-shot Target ID
Figure 8: Our zero-shot generation already achieves high success rates and quality. However, it oc- casionally struggles to preserve consistency accurately. Few-shot generation can produce results with better ID preservation.
Multi-modal movie scripts. The multi-modal script introduces face embedding better preserve consistency. Figure compelliggly demonstrates the effectiveness of this design. Specifically, the removal of face embeddings leads a significantly decreased ability of the model to preserve char- acter consistency. Face embeddings carry more nuanced and precise information compared to text alone. With face embedding, both the short-term and long-term consistency are well-preserved.
ID-preserving renderding. Before enabling ID-preserving rendering, our decoder has already shown the ability to reconstruct the target image. However, for images outside the training set, the reconstructed characters' appearances may differ slightly from the expected targets due the loss of fine facial features in the compressed tokens. After applying ID- ID-preserving rendering, our decoder exhibits significantly enhanced ability preserve character identities. The experimental results, illustrated in Figure 3 clearly demonstrate the effectiveness of the post-processing step.
Few-shot personalized generation. Our method serves as a powerful in-context learner, capable of generating results that are consistent with the style characters of the few references provided by the user. The results are presented in Figure Our model can produce results that are more consistent with the style and characters in the few-shot scenario.
CONCLUSION
We present MovieDreamer to address the challenge of generating long-duration visual content with complex narratives. This method elegantly marries the advantages of autoregression and diffusion, and is capable of generating very long videos. Additionally, we design the multi-modal script which aims at preserving character consistency across generated sequences. We further introduce ID- preserving rendering to better preserve character IDs and support few-shot movie creation due to the in-context modeling. This work potentially opens up exciting possibilities for future advancements in automated long-duration video production.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar arXiv:2308.1290 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023b.
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed movies: Story based retrieval with contextual embeddings. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomás Pa- jdla, and Jianbo Shi (eds.), Computer Vision - ACCV 2020 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 December 4, 2020, Revised Selected Papers, Part V. vol- ume 12626 of Lecture Notes in Computer Science, pp. 460-4777 Springer, 2020. doi: 10.1007/ URL http:://lll..rrr//000000///////0 0-000-000001001111-------00----1
978-3-030-69541-33 28.
28
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets. NONE, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision language-attiionn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 058223777.888.8 2023.
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL hnttps:///ppnaat
com/ research/
wi.do--generationnmoon. s-as--aasrrrrr 1 wwdddissmsieee
s iimiimmlaae imulators
Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 13642, 2024.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. arXiv preprint 1rXiv:2310.207000 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24118-24198, 2024.
An-Chich Cheng. Hongxu Yin. Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint AXiv:2406.01.884 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03378. 2023.
Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, and Nur Muhammad Mahi Shafullah. Robot utility models: General policies for zero-shot deployment in new environments, 2024. https:
//a //arxiv. wwwp:i..org/ab//2440020 05865
Zhangyin Feng, Yuchen Ren, Xinmiao Yu, Xiaocheng Feng, Duyu Tang, Shuming Shi, and Bing Qin. Improved visual story generation with adaptive context modeling. arXiv preprint arXiv: 2305. 168/1, 2023.
11

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. International Conference Learning Representations, 2024.
Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng. Ying Shan, and Qifeng Chen. Animate-a-stonee Storytelling with retrieval-augmented video generation. arXiv preprint arXiv: 2307.06940, 2023.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streaming(2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv: 2403,14773, 2024.
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018. URL.
https: / waw.iv.... otg/ alss 706. 08500
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik P.
Kingma, Ben Poole, Mohammad Norouzi, J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv: 22/0.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. 2022b.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-l: generative world model for autonomous driving. arXiv preprint arXiv: 2309. I 17080, 2023.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: 2210.03094, 2022.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.0922 2024.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hor- nung, Vighnesh Birodkar, Jimmy Yan, Ming-Chan Chiu, Krishna Somandepalli, Hassan Ak- bari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hen- don, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv: 2312.14125, 2023.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. arXiv preprint arXiv: 1.00922. 2018.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9559 2024.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv: 2. 17043, 2023a.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Pho- tomaker: Customizing realistic human photos via stacked id embedding. arXiv preprint arXiv: 23 2312,00 240411 2023b.
12

---

Published as a conference paper at ICLR 2025
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv: 2311.10122, 2023.
Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yanfeng Wang, and Weidi Xie. Intelli- gent grimm open-ended visual storytelling via latent diffusion models. arXiv preprint arXiv: 2306,00073, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek- vl: Towards real-world vision-language understanding. arXiv preprint arXiv: 2403.05525 2024,
Xichen Pan, Pengda Qin, Yuhong Li, Hui Xue, and Wenhu Chen. Synthesizing coherent story with auto-regressive latent diffusion models. arXiv preprint 100500 2022.
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng. Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv: 2310.02992, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv: 2307.01952, 2023.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He. Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv: 2310.15169, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. International Confer- ence on Machine Leurning, 2021.
Tanzila Rahman, Hsin-Ying Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. Make-a-story: Visual memory conditioned consistent story generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2493-2502, June 2023.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009-13011 2024.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan. Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li. Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024a.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26374-26388 2024b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba. Vicki Cheung, Alec Radford, and Xi Chen. Im- proved techniques for training gans, 2016. URL wttpt:tt//// /ttpppp/:aa.i..coooottt wwwp:iii.org//aaa hwapc//abs/6166 03498
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open lcrle-scale dataset for training next generation image-texi text models, 2022. URL
https: / rxviv.v... wtt./bo.//ooooooooooooo.oo/ooo./ 221 08402
13

---

Published as a conference paper at ICLR 2025
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Sitong Su, Litao Guo, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Make- sttoyy--toryy A general framework for storyboard with disentangled and merged control. arXiv preprint arXiv: 2312.07549, 2023.
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin. Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. arXiv preprint arXiv: 2307,00716, 2023a.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo. Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models in-context learners. 2023b.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv: 2307.05222, 2023c.
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consistent lext-to-image generation. arXiv preprint arXiv: 2402.03286, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXivv 2023.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. transformers, 2024.
Givt: Generative infinite-vocabulary
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. 2023a.
Fu- Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising, arXiv preprint arXiv: 2305. 2023b.
Wen Wang, Canyu Zhao, Hao Chen, Zhekai Chen, Kecheng Zheng, and Chunhua Shen. Au- tostory: Generating diverse storytelling images with minimal human effort. arXiv preprint arXiv:2311.11245 2023c.
Yi Wu, Ziqiang Li, Heliang Zheng, Chaoyue Wang, and Bin Li. Infinite-id: ddentityprrseervd personalization via id-semantics decoupling paradigm. arXiv preprint arXiv: 2403. I 1781, 2024.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar- image editing with diffusion models, 2022.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang. Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming. Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. Nuwa-xl: Diffusion over diffusion for extremely long video gener- ation. arXiv preprint arXiv: 2303. 12346, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv: 2403.15378, 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction- -tuned audio-visual language model for video understanding. Conference on Empirical Methods in Natural Language Process- ing, 2023. doi: (0.488500/axx1.2200.22222
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In arXiv preprint 12X:502212.04501, 2022.
14

---

Published as a conference paper at ICLR 2025
Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in visual- linguistic manner, 2022.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Con- sistent self-attention for long-range image and video generation, 2024.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Diffusion Autoencoder. We use the pretrained CLIP image encoder (Radford et al., 2021 to generate spatial tokens. The spatial tokens are compressed into two tokens using the com- pressor, a network with Transformer blocks. The architecture of the compressor is shown in Figure 2 First, we process the spatial tokens using three transformer blocks. We then inter- change the dimensions of the hidden states and compress them using a linear layer. The compressed hidden states are further processed through three transformers to obtain the final com- pressed token. We find that setting the compressed token to is sufficient for well reconstructing the image.
t: number of compressed tokens
256 1024
Spatial Tokens
Transformers
x
Hidden States
256 x 1024
Dimension Exchange
Subsequently. we employ the pretrained U-Net of SDXL (Podell al., (023) our decoder render the compressed tokens back into images. Notably, the original SDXL architecture demands an additional projected text embedding as input. To address this, we concatenate and project the compressed token into the desired input space of SDXL simply using linear layer.
Hidden States
1024 x 256
MLP
Hidden States
1024 x
Dimension Exchange
Compared to the image dataset, our collected movie dataset may lack diversity in all kinds of objects in the world, which somewhat limits the model's performance. To ensure model develops general understanding of various objects first, we employed a three-stage training strategy to train a more effecttive model. In the first stage, we fine-tuned the pre- trained U-Net and the compressor on the large existing image dataset resolution of 768 768 with cosine scheduling learning rate that ranges from le-4 to 2e-5 and an Adam W optimizer for 120k steps, allowing the model to understand various objects. Our dataset comprises images from OpenIm- ages (Kuznetsova et al.||2011 [2018), JourneyDB (Sun et al. 2023a), and Object365 (Shao et al. 2019).
Hidden States
t 1024
MLP
Q
K
V
x 2048
x 1024
t 1024
Transformers
x
Compressed token
In the second stage, we further fine-tuned the model using our self-collected movie dataset at resolution of 896 x 512. The model trained using AdamW with constant learning rate of 2e-5 for 120k steps in the second stage. For the first stage and second stage, we first freeze the decoder and only train the compressor for the first 20k steps, driving it to compress images more appropriately. To enable the model to generate results consistent with movie lighting, we further set the noise offset 0.05 during training.
Figure 9: Architecture of the com- pressor.
In the third stage, to better preserve character ID, we freeze the compressor and concatenate the compressed tokens with the text embedding and face embedding inputs the cross-attention and further finetune the decoder. Text embedding and face embedding are available in multimodal script data. Since the dimensions of the description embedding and facial embedding differ from those of the compressed token, we first project them to the same dimension as the compressed token using 2 two-layer MLPs with GELU as the activation function. Given that different frames have varying numbers of descriptions and face IDs, we pad the projected description embedding and projected facial embedding with zero tensors to extend the sequence length 15 and respectively. For instance, if frame has projected description embedding of shape [12, D] and projected facial embedding of shape [3, DJ. we pad them with zero tensors of shape [3. DJ and [2. D] to achieve shapes of [15, DJ and [5. DJ. Finally, we concatenate the compressed tokens with the projected description embedding and projected facial embedding, resulting in a [22, DJ tensor, which serves as the key and value input for the UNet cross-attention block. The decoder and the MLPs are trained using AdamW with a constant learning rate of 2e-5 for 120k steps and a noise offset of 0.05. The entire training process of the compressor and decoder takes 3 weeks with 6 NVIDIA A800 GPUs.
16

---

Published as a conference paper at ICLR 2025
Autoregressive Model. We utilize pretrained LLaMA 7B model as the backbone for our toregressive model, which is subsequently trained with our curated multi-modal script data. We use two-layer MLPs with GELU the activation function to unify different modalities into the input space of the large language model. We have three distinct modalities that need to be unified into the input space: compressed tokens, facial embeddings, and description embeddings. Therefore, three MLPs are trained alongside the autoregressive model. We set the length of the context frames to 128, which results in the max sequence length around 5000. The global autoregressive model is trained for 3 days using 4 NVIDIA H100 GPUs with constant learning rate of 2e-5 and 2k steps for warm-up.
To help the model distinguish the script for different frames, we inserted two special tokens, [START OF FRAME] and [END OF FRAME], the beginning and end of each frame's script, respectively. Furthermore, in our setting, there are three different modalities: face embedding, com- e
pressed tokens, and description embedding. The face embedding is extracted with FARL, (Zheng et al. 2022). and for each frame, we extract the top characters. The compressed tokens are generated by leveraging our trained encoder e, with the target frame as input. The description is divided into different sentences, and each sentence is encoded into one [CLS] token using Long- CLIP (Zhang et al. 2024). We treat the remaining text such as "Characters:" and "Scene Elements:" identifiers, which are tokenized the same LLaMA, shown in Figure ?
The identifier and descriptive texts in multimodal script. Here, we provide an example to help readers understand the multimodal script detail. Consider the example multimodal script in Fig- ure 2 [Characters:], [Scene Elements: J. [Plots:] and the symbol [-] are identifiers, while [A man black suit...). [A woman with a teal headdress..... [Crowded event], [Élegent Lighting), [A man woman.... ... are dancing closely...] are text descriptions. As shown in Figure 2 orange parts are identifiers, and blue parts are text descriptions. The identifier structures the input format, aiding the autoregressive model in understanding the different types of content within the script. The de- scription is responsible for providing detailed information about the Character, Scene Elements, and Plot.
ID- ID-preserving Rendering. To enhance our model's ability to maintain character consistency, we additionally input the description embedding encoded by LongCLIP (Zhang et al.) (024) and the face embedding extracted by FARL (Zheng et al., (022) into the trained decoder. Both the text embedding and face embedding are aligned with the decoder's input using two-layer MLPs and then concatenated with the compressed tokens to serve as the input for cross-attention.
We assume maximum of 15 sentences for description embedding and character IDs, If the inputs do not meet these quantities, we pad them with zeros. Moreover, for the concatenated embedding, we randomly mask each token with a probability of 0.2. This encourages the model to learn the relationship between facial features, text descriptions, and the compressed token.
Continuous Token Modeling, The k-mixutre GMM models the distribution p(c, M,), where e₁ is the continuous real-valued image tokens of sequence length S and dimension d. Her and M, refer to the previous history information and the current multi-modal script. The GMM consists of k multivariate Gaussian distribution components that are independent across the channel dimension d. To construct the GMM, we replace the logit prediction layer of our autoregressive model initialized using LLaMA with linear layer with 2kd + A dimension for output. Subse- quently, the mean m and the variance v of shape S x kd and mixture weights w of shape $ x k are derived. m consists of a stack of tensors [m] (2) ( ( (*),, ( Similarly, v = (2(1), v(2) .... (1)) The mixture weight is normalized along the channel dimension. Given the ground truth image embedding e, aim to minimize its negative lokeliiel within the GMM:
where m, is of shape S x d.
-log(p(e/H Mr)) log
II
W
II N (eLalm
( (
1=1
k
d
Σ
log
W
N
( m
1=1
17

---

Published as a conference paper at ICLR 2025
In a distant northern land, there was a tribe renowned for its berserkers, known for their courage and fearlessness.
Aron was always at the forefront: he was both the chief and the fiercest warrior.
The berserkers spread war across many lands.
However, one day. Aron discovered that their former victims had united seek vengeance against their tribe.
The tribe members all became aware of the sins they had committed in the past.
The war inflicted immense losses on their tribe.
His wife, Eleanor injured seriously during escape. urged him to see that war was merciless, with no victors ooll suffering.
Knowing Aron't 's stubborn nature, Eleanor lay down in disappointment.
Nevertheless, Aron clung his previous strategies.
Elcanor's injury and the prolonged suffering eventually made Aron realize that Eleanor was right. He struggled sleep cach night.
After much reflection, he decided to leave the tribe and embark on journey of atonement alone.
One morning, he stared into the distance, deep in thought.
Figure 10: Additional Zero-Shot results.
18

---

Published as a conference paper at ICLR 2025
Many famous big would attend the parties, but people didn't 't know who Gatsby really
Gatsby often held incredibly lavish parties at his home.
However, Nick received an invitation to Gatsby's 's party,
Nick agreed to help Gatsby, so Gatsby had many people meticulously decorate Nick's house.
At the party, he met Gatsby, who wanted rekindle his lost love Nick's cousin, Daisy.
the appointed time approached, Gatsby became very nervous.
Seeing Daisy, Gatsby's suppresep feelings burst forth.
Daisy was also thrilled to Gatsby. They reignited their past romance.
Daisy arrived as promised.
Gatsby made Daisy and her hushand, Tom, into room confrontation, hoping Daisy would leave with him.
However, Tom had already investigated Gatsby's 's background. He began to publicly expose Gatsby.
At first, Gatsby managed to remain calm.
But Tom went on to reveal many unsavory facts about Gatsby.
As result, Gatsby completely flew into rage.
Witnessing this, Daisy also broke down tears.
Gatsby still wanted Daisy to lcave with him.
But Daisy was already in distress and ran out of the crowd.
Tom stood there smiling, knowing he had won.
Figure 11: Additional Zero-Shot results.
19

---

Published as a conference paper at ICLR 2025
Andy wrongfully convicted because he was suspected murder.
The lawyer didn't it believe in Andy's 's innocence and sent him to prison.
Mcanwhile, Red was also undergoing parole review.
TLAA VOOIE
PEOTHRE TO
EVEE TI/EPE
SHO nv
But unfortunately, his parole was denied.
After entering prison, Andy and Red formed connection.
Red found Andy interesting and they quickly became friends.
Ignoring the guard's objections, Andy seized the chance offer help to the sheriff.
One day, Andy and his friends were working outside.
Andy heard the sheriff worrying about economic issues.
As result, Andy won beer for his friends.
The warden heard of Andy and had him work for him.
Andy agreed.
Andy took this opportunity to improve the quality of the library the prison. Inmates' quality of became better.
But the warden was no longer willing to let Andy leave prison because he knew too much about the warden's dirty work.
Andy told Red look for something he left under a tree after his release.
But Andy eventually succeeded in escaping from prison.
Soon after. Red's parole was also granted.
Initially, Red took this joke.
Ale & Gost detre t
fioe de thn far bole
He found the oak tree that Andy had mentioned.
Beneath the tree, he discovered the letter Andy had left for him.
Red was happy because he knew they would meet again.
Figure 12: Additional Zero-Shot results.
20

---

Published as a conference paper at ICLR 2025
Gerald is a math professor, and one day he left difficult problem the blackboard.
Will is a janitor on campus, and he noticed the problem the professor left behind.
Meanwhile, Will is a genius in mathematics, and he quickly became immersed in thought.
1
0
A1
B
6
0
01
0
0
8
8
8
01
8
But after returning home, Will pondered solution in front the mirror.
At night, Will would go to the har to hang out with his friends.
The next day, Will left the answer the blackboard.
The professor was very surprised, but no students claimed they solved it themselves.
One day, a student troubled Will's friend at the bar.
But Will managed to refute him with his talent.
However, Will is a troubled youth. He was taken to court for fighting.
This helped him Skylar's favor.
The judge intended send him to prison.
The professor alrcady aware Will's situation and proposed that he study mathematics and undergo regular psychological evaluations so he wouldn't have go to prison.
But many psychologists are insulted by Will. So Gerald had turn to his old friend Sean for help.
At first, Sean also found it difficult to cope with Will.
But he didn't give up. His sincerity eventually touched Will.
In the end, Will achieved redemption with Sean's 's support.
After that, Will embarked on new journey in life.
Figure 13: Additional Zero-Shot results.
21

---

Published as a conference paper at ICLR 2025
James is checking the spacecraft system.
This is his first mission. He is kind of nervous.
James launches the spaceship. They are about to land on new planet.
Mcanwhile, the command center Steven believes that this planet is However, others disagree with
is debating whether they should worth exploring. land on this planet.
Steven's idea, as the planet is full
of unknowns and could pose dan-
gers.
But Steven believes that high risks come with high rewards, and he stands by his opinion.
They have heated debate.
In the end, Steven convinces cryone and sends the landing in- structions to the spaceship.
James and the astronauts accom- panying him receive instruc- tions and begin preparing for as they prepare for landing. landing.
Steven is directing James and the The spaceship gradually ap- others the command center proaches planet's surface.
Figure 14: Additional Zero-Shot results.
22

---

Published as a conference paper at ICLR 2025
KeyFrame
Ours
Story Diffusinnn
StoryGen
134
2230
2234
2652
3303
3308
3721
3722
3976
4479
5228
Figure 15: Additional story comparison results.
23

---

Published as a conference paper at ICLR 2025
During training, additionally introduce the { and lg loss between the predicted image tokens and the ground truth tokens to improve the model's performance. However, we find that these losses are not on the same scale the negative log- likeeiii.oood. The negative log- lekellkli is way larger than the l and &2 loss, which reduces the influence of the & and l loss during the optimization. The difference in scale is because the constant term and covariance matrix of the negative log- lieeikkd are directly related to the dimension d. Concretely, for variable x with mean p. the calculation of the negative log-likelihood is essentially represented as:
(8) where A 1og)(2)) is the constant term and Σ is the covariance matrix. As increases, both the constant term and the size of the covariance matrix grow, resulting in significant increase in the loss value. To bring the different losses the same scale, we scale the negative log-likelihood loss by dividing it by the dimension d. Consequently, the negative log-likelihood loss in Section 3.4 is formulated as:
NLL(X; Σ) 1og(2x) log ز
2 +
11) E
(x p))
Lull
B(- ([-log(p(e/HHi.. M.))
T
log eeg ecc de))
[=1 i=1
h
d
E
log
Σ
wi
(6) MeLalm 410)
B EVALUATION METRICS
We propose new metries evaluate short-term (ST) consistency and long-term -erm-ttemmmn (LT) consistency. For short-term consistency, we select all the frames where a character appears. We then calculate the similarity of CLIP embedding of selected consecutive keyframes, as consecutive keyframes where the same character appears usually have high similarity. For long-term consistency, We calculate the proportion of generated frames where the character appears to the ground truth frames where the character appears. For example, if the GT frames where a character appears are 1, 2. 3, 4. 5, and the generated frames where the character appears are 1, 2, 3, then the long-term consistency is calculated as 3/5. To select the frames where character appears, we first select an image of the main character and use FARL to obtain the character's 's embedding. We then compute the similarity of this embedding with FARL embeddings of all other frames and select images above a certain threshold.
The Aesthetic Score (Schuhmann et al. 2022) is calculated based on the CLIP image features. Each image is first passed through the CLIP image encoder to obtain an embedding. This embedding is then fed into an additional small neural network, which predicts an aesthetic score.
C MULTI-MODAL SCRIPTS DATA PIPELINE DETAILS
Multimodal script construction. In this chapter, we explain in detail how we construct the mul- timodal script data. Specifically, We collect a massive movie dataset, with movies coming from Condensed Movies (Bain et al. 2020) and others collected using the same methodology as Con- densed Movies. The data consists of around 23,000 long videos, which subsequently results in million keyframes with corresponding script annotations. The overall multimodal script synthesis is as follows:
Extract keyframes from long videos at I-second intervals, removing the beginning and end segments. These segments primarily contain sponsor messages, acknowledgments, and ad- vertisements, lacking useful information. Subsequently, center-crop and resize each frame to width of 896 pixels and a height of 512 pixels. Finally, leverage Vision Language Model (Lu et al. 2024) to generate descriptions for the entire image.
Using GrO (Ren et al., 2024a) with "Person" as the prompt, extract the top 5 characters from each frame. If the number of characters exceeds 5, the extra characters are ignored. If the number of characters is less than 5, there is additional processing.
24

---

Published as a conference paper at ICLR 2025
Face ID
A mon black suit with: slicked-bacc bbckk hair. -IMG>
Characters:
netrieval
man in a black suit with slicked-bor soodddbck hair. e woman with a teal headdress and matching necklace.
man in woman in dancing closely crowded event. The The The background filled with other people...
Multimodal
Scripts Fusion
Scene Elements:
Crowded event
Elegant Lighting
Plot: mon woman ore
Face
dancing
closely at crowded event.
with headdress and matching necklace, <IMG>
retrieval
Multi-modal scripts data pipeline. We generate image captions using existing VLMs, and then further utilize LLMs to reformat the annotations into structured script. We use the face retrieval method to extract different face embeddings of the same character, which benefits anti- overfitting data augmentation in Section 3.4
Figure 16:
Employ an existing Vision Language Model (Lu et al. 2024) to obtain captions for each segmented character. The face of each character extracted construct the face ID bank.
Use Large Language Model, taking the captions for the entire image extracted step and captions for each character in step 3 as input, to derive Scene Elements and Plot, and multimodal script format. At this point, we have structured
ask the model output Scene Elements and Plot but still lack Character. We then format all the character captions corresponding to each frame into the Character section of the multimodal script.
However, the character face IDs for each frame are derived from the same frame, which can lead the model overfitting by directly copy copy-pasting characters into the generated results. To address this, we use FARL (Zheng et al. 2022) to randomly retrieve the same character's ID from the face ID bank input. This strategy effectively mitigates the overfitting issue.
D ADDITIONAL RESULTS
In this section, present additional generated stories and videos. Our method exhibits strong generalization capabilities, producing high-quality results with inputs not included in the training set. Our method can understand text and character inputs, capable of generating results that align with the storyline while ensuring character identity consistency. The overall scene consistency is also well- peelereee,,,dd as our autoregressive model able to leverage context information as the reference. Most importantly, we the first method to generate extremely long contents while preserving both short-term and long-term consistency. Besides, we even ensure that the IDs of different characters remain consistent before and after camera change.
Generation process. Our autoregressive model is similar to multi-turn dialogue model. Specif- ically, we input the multimodal script corresponding to a keyframe, and the model autoregressively outputs two compressed tokens. Then, based this, input the multimodal script for the next frame to obtain the next two compressed tokens, without losing historical information. After ob- taining all the compressed tokens, we input them along with the face embeddings and description embeddings from the multimodal script into the decoder to generate the keyframe. The keyframes subsequently leveraged by maage-to-vddes model generate videos.
Additional results. Our method effectively preserves both short-term and long-term consistency for multiple characters. We present numerous results in Figure I0 II] 12 13 and [4] Our method is able preserve multiple characters within long contents without the need for any fine-tuning.
25

---

Published as a conference paper at ICLR 2025
Step1: Frame Caption
Step3: LLM Fusion
Step1 input:
Step3 User Input:
Transform the provided description {Step1 VLM Output) and its characters(Step1 VLM Output (List) into structured format by identifying Soene Elements, and Plot. Ensure to replace placcholders within brackets 'I with specific details from the text. Character descriptions should be simple, without detailed elaboration. The plot summary should succinctly encapsulate the central event or emotion more than 50 words. Follow the structure below for formatting
Provide detailed yet. concise description the image that allows DALL-E Midjoumey recreate forusing on key elements such subjects, background, and objects.
your output:
Describe colors, textures, lighting, and spetial relationships between objects, ensuring accuracy. If there is motion, vividly depict its direction and nature. Aim for a clear, comprehensive description, but avoid unnecessary length.
Somne Elements:
[D DDsscrittion of First key element scene]
[D [Descniption of Second key element of the scene)
[Description of Third key element of the scene]
Step1 VLM Output:
Plot:
In warmly lit room, boy in plaid shirt and glasses stand facing each other The man holds a pen, poised to write on a piece of paper, while the boy examines letter with crest on it. Behind them, a woman a blue shirt glasses Joans against wooden cabinet, observing the scene.
brown suit and young
[A concise summary focusing on the character event emotion, within 60 words).
Step3 LLM Output:
Scene Elements:
Warmly room
Man holding
Step2: ID Caption
A letterwith a crest
Step2 User Input:
Plat:
and a young boy stand facing each other. The man holds pen, poised to write piece of papor while the boy examinos letter with a crost on it. Behind them, a woman observing the scene.
warmly lit rom,
Step Compose
Stepá Output:
Provide dotailed concise description of the character that allows DALL-E or Midjoumey recreate it, focusing on key elements such as subjects, and objects. D Describe colors, textures, lighting, and spatial relationships between objects, ensuring accuracy. If there is motion, vividly depict its direction and nature. Aim for cleat, comprebensive description, but avoid unnecessary length.Better Better restrict in 100 words
Characters:
image man brown suit tie, appearing supprsed, in indoor setting with blury background.
<IMG>
young man in plaid shirt reads letter with an owl stamp, in a room with woodon cabinets. <IMG>
Scene Elements:
Warmly lit mom
Step2 VLM M. Output (List)::
Man holding pen
Image of a man in brown and appearing
A letter with crest
surprisod, indoor setting with bluny background.
<IMG>
Plot:
warnly lit room, and young boy stand facing each other The man holds pen, poised to write on piece of paper, while the boy examines letter with a crust on it. Behind them, woman observing the scene.
young man in a plaid shirt reads letter with an owl stamp, in with wooden cabinets, <IMG>
Figure 17: The overall data pipeline and corresponding prompts we use.
Even when the scene changes, our method effectively maintains the same character ID even during camera switches. Due to space limitations, we only select certain excerpts showcase here. Please refer to the TITANIC generated by our MovieDreamer in supplementary material for longer results.
ID preservation. We showcase our ability to preserve character ID in Figure [8 Our method ex- hibits strong generalization capabilities, allowing to generate target characters without any further training.
Video results. The generated keyframes serve as the condition to generate videos. We show the results generated with our improved video generation model in Figure Our keyframes have
26

---

Published as a conference paper at ICLR 2025
Figure 18: ID Preservation results. Our method demonstrates the great ability to preserve ID. The compressed token for each keyframe is generated with the corresponding multimodal script as input, and the keyframes are rendered with compressed tokens, face embedding and description embedding input. We show the target ID the left.
successfully preserved both short-term and long-term temporal consistency, facilitating the seamless rendering of these frames into a cohesive and harmonious video.
Again, would like highlight that our primary contribution lies in the hierarchical controllable very long generation of visual contents using MLLM for keyframe generation and diffusson model for video rendering. Any high-quality image-to-vins ti-icooo model can be integrated with approach. We showcase video results generated using higher-quality image-to-vide keyframes input in Figure 20 which strongly demonstrates the effectiveness and flexibility of our method. We believe our approach is highly promising has immense potential.
models with our generated
Different shot types and styles. We find that although did not explicitly model the camera shot types, the model can implicitly learn various shot types based on the multimodal script. The model is capable of selecting the most appropriate shot type based the input. The shot types of anchor frames further assist the image-to-vine video model in generating videos with various shot types. Our method is also capable of generating cartoon results. We showcase the results in Figure 21
E ADDITIONAL EXPERIMENTS
E. ADDITIONAL COMPARISONS
Our method effectively preserves both short-term and long-term consistency, maintaining the overall style and character identity. Other methods ex- hibit increasing inconsistencies the number of frames grows, such changes in character ap-
We showcase additional comparisons in Fig 15
27

---

Published as a conference paper at ICLR 2025
Figure 19: Example video results with our video generation method.
Figure 20: Example video results using existing better image-to- model LUMA. Existing image-to- viveeeo models can be integrated with our method produce high-quality ultra-long videos, which demonstrates the flexibility and effectiveness of our MovieDreamer.
pearance and shifts in style. Our method can generate extremely long content while nearly perfectly maintaining the consistency of multiple characters throughout.
E.2 ADDITIONAL ABLATIONS ON THE NUMBER OF COMPRESSED TOKENS
Our goal and an important tradeoff. While more tokens may yield better results, they sacrifice the context length of autoregression. Since aim for long video generation, using as few tokens as possible a reasonable design choice. Our goal is to achieve the highest quality images with minimal number of tokens.
28

---

Published as a conference paper at ICLR 2025
Close-up
Medium Shot
Long
Aenil Shot
Over Shouldar Shot
Cowboy Shot
(b)
ttmppiati Tature plans
Anya read Load's mino bicoamin very
Anya
proce sang Angla gets same good Anya coing
agress Anye's
Figure 21: (a) Although our method does not explicitly model shot types, it can still implicitly learn various shot types and generate high-quality results with suitable shot types, demonstrating the potential of our approach. (b) Our method can also generate cartoon results and interleaved characters.
Table 3: Quantitative ablation study on the number compressed tokens.
Method
CLIP-simt
Inceptiont
Aesthetict
FID4
2 tokens
0,825
8,235
6.378
0,406
tokens
0.841
8.193
6,457
0,348
Why do we choose 2 tokens. Existing work Paint-by Paint-by-Exxe by-Example (Yang et al. 2022) demonstrates that a single token can effectively reconstruct the original image. Building on this discovery, we added a few additional tokens. We compared the reconstructed images using 8 tokens and 2 to- kens, with the metrics shown the accompanying table below. The metrics indicate that increasing the number of tokens does not offer significant improvements in results. As shown in Table 3 calculate the inception score, aesthetic score as well as FID score of images of different numbers of tokens. We further compare the CLIP similarity between reconstructed images and target im- ages. The evaluation metrics demonstrate that the results considering objective of long content generation, this slight improvement is not enough for us to prioritize when making trade-offs regarding sequence length and our limited computational re- sources, as tokens will lead to a much shorter sequence length and demand more computational overhead. Consequently, we eventually chose 2 tokens. But we believe that if there are enough computational resources, 8 tokens will be better choice.
tokens are slightly improved. But
E.3 ADDITIONAL ABLATIONS ON CONTINUOUS TOKEN SUPERVISION
We conduct additional experiments further explore the importance continuous token supervi- sion as well as the L1 and 1.2 losses. The evaluation metrics are shown in Table 4 As visualized Figure 22 when the model is trained using only continuous token supervision as the objective function, it tends to produce poor results with characters placed at the edges of the image. However, this problem effectively resolved when and L2 are introduced as additional objective functions during training, Our experiments demonstrate that both continuous token supervision and the L1 and L.2 losses play crucial roles in optimizing the model.
Table 4: Quantitative comparisons of results trained with different losses.
Method
CLIP
Inception
Aesthetic
FIDL
STG
LT
Con-LI-L2
19,584
9,698
6,093
2.043
0.646
0.814
Con-LI-L2-ref
20,071
9.842
6,288
1.912
0.701
0.893
L1-L2
18.833
8.609
5,893
2.714
0.606
0.755
LI-L2-ref
19.431
8.774
5.979
2.560
0.616
0.776
Con
16.882
7.463
5.352
3,283
0.576
0.701
Con-ref
17.039
7,580
5,411
3.136
0,571
0.708
29

---

Published as a conference paper at ICLR 2025
Thiget
LETE ONL
Con Only
Con +
Comparison between using only continuous token supervision and incorporating both L1 and 1.2. The results show that relying solely on continuous token supervision leads the model to place characters at the edges of the image, resulting in low-quality keyframes. With L1 and L.2 only, the model able to learn layout information but struggle to maintain target ID. With continuous token supervision, LI and L2, the model is able to generate results with reasonable layout and preserve the target ID effectively.
Figure 22:
Figure 23: When there are too many characters in frame, the model sometimes confuses the character IDs.
F LIMITATIONS AND FUTURE WORKS
Despite our method demonstrating strong potential in generating long sequences, still has some limitations. (1) We employ LongCLIP to compress each sentence of descriptive text into a single token. While this strategy effectively reduces the sequence length, provides only a coarse rep- resentation of each sentence, which causes information loss. (2) Our method's ability to preserve object consistency is limited. We did not construct the multi-modal script with scene information, as the costs far exceed our capacity. Moreover, shown in Fig, 23 if there are too many people in one frame, our method is prone to mixing the appearance of characters. (3) Analogous to other large models, approach requires substantial data and computational power for training. (4) Cur- rently, we generate 128 keyframes at a time and run multiple iterations of this process to create long content. However, goal is extend the maximum token length, fundamentally addressing the
30

---

Published as a conference paper at ICLR 2025
long sequence generation by using long sequence generation methods in LLMs. Due to resource constraints, we are unable to pursue this approach at present and leave in future work.
Social impact. Our method can generate high-quality long stories and videos, significantly lower- ing the barrier for individuals to create desired visually appealing content. However, it is essential to address the negative potential social impact our method. Malicious users may exploit its capabil- ities to generate inappropriate or harmful content. Consequently, it is imperative to emphasize the responsible and ethical utilization of our method, under the collective supervision and governance of society as a whole. Ensuring appropriate usage practices requires collaborative effort from var- ious people, including researchers, policymakers, and the wider community. Furthermore, the code, model, well as the data, will be fully released to improve the development of related fields.
This work was supported by the National Key R&D Program of China (NO.20222D0110100) and the National Natural Science Foundation of China (No. 62206244).
Acknowledgements
31


Reviews:
[{"review_id": "gRV4FUbj40", "reviewer": "Reviewer_zniU", "summary": "This paper presents MovieDreamer to make long video. This is challenging because current video diffusion models can only deal with short clips and do not have character consistency. The model trains a MLLM that can predicts the visual tokens with movie script conditioning. Then using the diffusion decoder to decode the condition into key frames, and using I2V to render it into video clips.", "questions": "Since authors are changeing LLaMA input to CLIP embedding (for sentence), does it cause much for the LLM to adapt to such input?\nHow much cherry-pick do you need for a single long video?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The quality is GREAT!\nEngineering wise, this paper produces a product-level solution to long-video (movie) generation; Scientificly, this paper also explores training a MLLM to output visual features given the input scripts, instead of simply using an agent framework to compose scripts, and use subject-driven models to preserve character identity.\nThe ID Preserving is also very effective", "weaknesses": "Lack of non-human results, don't know if such methods can generalize well to general domain long-vedio generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NElmGepPNE", "reviewer": "Reviewer_xVaX", "summary": "This paper presents a hierarchical generation framework for long video sequence generation, which can be potentially applied to movie video generation. The key idea is to leverage the autoregressive models to generate key frame tokens, and then use diffusion models to decode the tokens into RGB frames. This approach leverages the advantage of any-length future prediction ability of AR model and the powerful rendering ability of diffusion models. Through some other techniques such as multimodal script, face embedding condition, etc., the framework can generate ultra-long video sequences with better quality.", "questions": "Please provide the evaluation of the ID consistency of the model, clarify the technical contributions, and discuss the limitations of the framework.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The model provides an effective integration of autoregressive model and diffusion models, for the long visual sequence generation task. The combination is reasonable by successfully leveraging the arbitrary-length generation capability or AR model and the powerful rendering ability of diffusion models. \n+ The utilization of the gaussian mixture model for the continuous visual token prediction is reasonable and inspiring.", "weaknesses": "- The paper proposes a lot of techniques to improve the identity consistency among different clips, however, I cannot find any ID-related evaluation on the generation results. \n- The paper did not analyze the boundaries of the proposed method. For example, \n    - when will the model fail at ID consistency since the model only provides one face embedding as the condition. \n- The technical novelty is relatively limited. There exist many multimodal generative models can do the similar key frame generation task, though they may not be directly used for movie key generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFTT9yfmir", "reviewer": "Reviewer_4ibH", "summary": "Existing diffusion model-based video generation models can only support short video generation and lack understanding of narrative structures and plot progressions. Therefore, this paper explores long-story video generation by leveraging the inference capability of autoregressive generation and video diffusion models’ rendering capability. Specifically, this paper proposes a hierarchical framework and a structured multimodal story input representation. The framework fine-tunes the multimodal large model LLAVA to generate compressed tokens of keyframes and fine-tunes SDXL as a decoder to generate keyframe images from the compressed tokens. Finally, SVD is used to generate the video. Additionally, the paper proposes a few-shot training method, enabling the model to achieve customized generation. Extensive experiments compare the proposed method with existing story keyframe generation methods, demonstrating superior performance.", "questions": "I have outlined some concerns and suggestions in the weaknesses section, and there are additional questions that need clarification.\n\n1. Is the covariance matrix in Eq. 7 of the supplementary material a diagonal matrix?\n\n2. In the few-shot training section, what is the “episode’s visual tokens” (line 302)?\n\n3. In Fig 8, what are the reference images for generating the few-shot results?  If I understand correctly, face embedding is also used as part of the multimodal script to generate zero-shot results. For the few-shot generation, what images are used as references, and how do these reference images help improve identity consistency?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed framework seems promising for generating long video. \n2.\tThe proposed method can effectively generate character-consistent story keyframes, and the experimental results show the superiority of the proposed method to other state-of-the-art methods.", "weaknesses": "The overall writing of the paper is relatively clear, and the experimental results are promising. However, some of the descriptions appear to be incorrect or misleading, and the experiments are also not entirely comprehensive.\n\n1. In line 242, why are the parameters of the GMM stated as 2kd means and 2kd variances? According to the reference work, each GMM parameter should contain kd means, kd variances, and k coefficients for each compressed token.\n\n2. What would the results look like if Eq. 2 were used as the sole objective function? There doesn't appear to be a corresponding analysis of this in the experiments. Is the inclusion of L2 and l2 necessary, or have previous works discussed the necessity of incorporating these terms?\n\n3. In Fig 2, if I understand correctly, the multimodal script in the top-left corner is used to generate a single keyframe. However, the three keyframes shown on the right under the VLM section could easily give the impression that they were all generated by the same script. This may lead to some confusion for readers. Additionally, according to Equation 1, the input to the decoder should also include a noise latent. However, the inference process shown at the bottom of Figure 2 does not fully display the complete inputs and outputs, which could lead to some misunderstandings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6VCv8YQbwZ", "reviewer": "Reviewer_Jjeo", "summary": "The paper proposed a novel hierarchical video generation framework MovieDreamer, which is able to generate long-duration videos with high visual fidelity. The framework consists of a SDXL-based visual autoencoder and a multimodality VLM. The autoencoder focuses on obtaining the visual tokens of input images, as well as rendering targeted keyframes, while VLM autoregressively predicts visual tokens. Quantitative evaluation shows that the proposed method outperforms sota in long video generation.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework propose a hierarchical way to combine autoregressive modeling and diffusion image generator together for long-term complex video generation.\n2. The paper proposed several novel techniques, e.g., continuous token supervision, anti-overfitting strategies, id-preserving, to improve the performance of the framework.\n3. The proposed method achieved state-of-the-art performance on quantitative evaluation.", "weaknesses": "1. L212: Eq1, the DAE has not been clearly defined in the paper. Also, in Fig2, I could not find DAE in the pipeline. I hope the authors could clarify this.\n2. It is a bit confusing whether Eq1 is the real training loss or not? According to Sec3.3 and Fig2, the input of SDXL should be image token, face embedding and description embedding, which does not align with Eq1. It is better to clarify the final training loss.\n3. It is not clear how random masking strategies work during training and inference stages.\n4. The high dropout rate (50%) seems to be unusual. Are all the parameters updated in VLM? Have the authors tried only tuning part of the network?\n5. L323: if the feature of the first frame is always used, will generated character be limited to small motion, and limited content? For example, for large rotation and movement, the performance of the proposed method may drop?\n6. What are the frame numbers of full-length video?\n7. From Fig8, while compared to zero-shot setting, the few-shot setting indeed improves the ID-preserving ability. However, there is still an obvious gap between target ID and generated image. What could be the potential reason for this? Any solution?\n8. What is the training cost? And what are the inference time and GPU memory requirements?  \n9. For quantitative evaluation, it seems that the reported metrics all focus on image level. They can not evaluate how good the performance of spatio-temporal modeling. It is better the authors could evaluate the proposed method using other metrics for video generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)