MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences

Authors: Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang, Hao Chen, Bo Zhang, Chunhua Shen
Primary area: generative models
Keywords: Generative Model, Vison Language Model, Long Video Generation, Long Story Generation

Abstract:
Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730556710545

thread_type:
official_review

<thread_object>
creator: reviewer_jjeo

<content>
summary:
The paper proposed a novel hierarchical video generation framework MovieDreamer, which is able to generate long-duration videos with high visual fidelity. The framework consists of a SDXL-based visual autoencoder and a multimodality VLM. The autoencoder focuses on obtaining the visual tokens of input images, as well as rendering targeted keyframes, while VLM autoregressively predicts visual tokens. Quantitative evaluation shows that the proposed method outperforms sota in long video generation.

strengths:
1. The proposed framework propose a hierarchical way to combine autoregressive modeling and diffusion image generator together for long-term complex video generation.
2. The paper proposed several novel techniques, e.g., continuous token supervision, anti-overfitting strategies, id-preserving, to improve the performance of the framework.
3. The proposed method achieved state-of-the-art performance on quantitative evaluation.

weaknesses:
1. L212: Eq1, the DAE has not been clearly defined in the paper. Also, in Fig2, I could not find DAE in the pipeline. I hope the authors could clarify this.
2. It is a bit confusing whether Eq1 is the real training loss or not? According to Sec3.3 and Fig2, the input of SDXL should be image token, face embedding and description embedding, which does not align with Eq1. It is better to clarify the final training loss.
3. It is not clear how random masking strategies work during training and inference stages.
4. The high dropout rate (50%) seems to be unusual. Are all the parameters updated in VLM? Have the authors tried only tuning part of the network?
5. L323: if the feature of the first frame is always used, will generated character be limited to small motion, and limited content? For example, for large rotation and movement, the performance of the proposed method may drop?
6. What are the frame numbers of full-length video?
7. From Fig8, while compared to zero-shot setting, the few-shot setting indeed improves the ID-preserving ability. However, there is still an obvious gap between target ID and generated image. What could be the potential reason for this? Any solution?
8. What is the training cost? And what are the inference time and GPU memory requirements?  
9. For quantitative evaluation, it seems that the reported metrics all focus on image level. They can not evaluate how good the performance of spatio-temporal modeling. It is better the authors could evaluate the proposed method using other metrics for video generation.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622137355

thread_type:
official_review

<thread_object>
creator: reviewer_4ibh

<content>
summary:
Existing diffusion model-based video generation models can only support short video generation and lack understanding of narrative structures and plot progressions. Therefore, this paper explores long-story video generation by leveraging the inference capability of autoregressive generation and video diffusion models’ rendering capability. Specifically, this paper proposes a hierarchical framework and a structured multimodal story input representation. The framework fine-tunes the multimodal large model LLAVA to generate compressed tokens of keyframes and fine-tunes SDXL as a decoder to generate keyframe images from the compressed tokens. Finally, SVD is used to generate the video. Additionally, the paper proposes a few-shot training method, enabling the model to achieve customized generation. Extensive experiments compare the proposed method with existing story keyframe generation methods, demonstrating superior performance.

strengths:
1.	The proposed framework seems promising for generating long video. 
2.	The proposed method can effectively generate character-consistent story keyframes, and the experimental results show the superiority of the proposed method to other state-of-the-art methods.

weaknesses:
The overall writing of the paper is relatively clear, and the experimental results are promising. However, some of the descriptions appear to be incorrect or misleading, and the experiments are also not entirely comprehensive.

1. In line 242, why are the parameters of the GMM stated as 2kd means and 2kd variances? According to the reference work, each GMM parameter should contain kd means, kd variances, and k coefficients for each compressed token.

2. What would the results look like if Eq. 2 were used as the sole objective function? There doesn't appear to be a corresponding analysis of this in the experiments. Is the inclusion of L2 and l2 necessary, or have previous works discussed the necessity of incorporating these terms?

3. In Fig 2, if I understand correctly, the multimodal script in the top-left corner is used to generate a single keyframe. However, the three keyframes shown on the right under the VLM section could easily give the impression that they were all generated by the same script. This may lead to some confusion for readers. Additionally, according to Equation 1, the input to the decoder should also include a noise latent. However, the inference process shown at the bottom of Figure 2 does not fully display the complete inputs and outputs, which could lead to some misunderstandings.

questions:
I have outlined some concerns and suggestions in the weaknesses section, and there are additional questions that need clarification.

1. Is the covariance matrix in Eq. 7 of the supplementary material a diagonal matrix?

2. In the few-shot training section, what is the “episode’s visual tokens” (line 302)?

3. In Fig 8, what are the reference images for generating the few-shot results?  If I understand correctly, face embedding is also used as part of the multimodal script to generate zero-shot results. For the few-shot generation, what images are used as references, and how do these reference images help improve identity consistency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685617113

thread_type:
official_review

<thread_object>
creator: reviewer_xvax

<content>
summary:
This paper presents a hierarchical generation framework for long video sequence generation, which can be potentially applied to movie video generation. The key idea is to leverage the autoregressive models to generate key frame tokens, and then use diffusion models to decode the tokens into RGB frames. This approach leverages the advantage of any-length future prediction ability of AR model and the powerful rendering ability of diffusion models. Through some other techniques such as multimodal script, face embedding condition, etc., the framework can generate ultra-long video sequences with better quality.

strengths:
+ The model provides an effective integration of autoregressive model and diffusion models, for the long visual sequence generation task. The combination is reasonable by successfully leveraging the arbitrary-length generation capability or AR model and the powerful rendering ability of diffusion models. 
+ The utilization of the gaussian mixture model for the continuous visual token prediction is reasonable and inspiring.

weaknesses:
- The paper proposes a lot of techniques to improve the identity consistency among different clips, however, I cannot find any ID-related evaluation on the generation results. 
- The paper did not analyze the boundaries of the proposed method. For example, 
    - when will the model fail at ID consistency since the model only provides one face embedding as the condition. 
- The technical novelty is relatively limited. There exist many multimodal generative models can do the similar key frame generation task, though they may not be directly used for movie key generation.

questions:
Please provide the evaluation of the ID consistency of the model, clarify the technical contributions, and discuss the limitations of the framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780999570

thread_type:
official_review

<thread_object>
creator: reviewer_zniu

<content>
summary:
This paper presents MovieDreamer to make long video. This is challenging because current video diffusion models can only deal with short clips and do not have character consistency. The model trains a MLLM that can predicts the visual tokens with movie script conditioning. Then using the diffusion decoder to decode the condition into key frames, and using I2V to render it into video clips.

strengths:
The quality is GREAT!
Engineering wise, this paper produces a product-level solution to long-video (movie) generation; Scientificly, this paper also explores training a MLLM to output visual features given the input scripts, instead of simply using an agent framework to compose scripts, and use subject-driven models to preserve character identity.
The ID Preserving is also very effective

weaknesses:
Lack of non-human results, don't know if such methods can generalize well to general domain long-vedio generation.

questions:
Since authors are changeing LLaMA input to CLIP embedding (for sentence), does it cause much for the LLM to adapt to such input?
How much cherry-pick do you need for a single long video?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732014332069

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
First of all, we greatly appreciate all the reviewers' thoughtful comments!

**Our Technical Contribution**

We are delighted to see that our work is recognized by many reviewers. Here, we would like to take this opportunity to reiterate our contributions:

- Addressing an Under-Explored Task: We are the pioneer to tackle the significantly under-explored ultra-long video generation. Unlike most existing methods that extend videos through looping or simple motion, our approach fundamentally achieves greater temporal extension while maintaining consistency across multiple characters.

- Unify Story Generation and Video Generation: Our approach unifies story generation and video generation in a hierarchical way.

- Flexibility of Our Approach: Our method is compatible with any existing image-to-video generation method and benefits from state-of-the-art image-to-video methods. This further highlights the flexibility of our approach.

- Structured Multimodal Scripts: We design the structured multimodal scripts to enhance the controllability and introduce a multimodal scripts data pipeline tailored for long videos.

- Supervision: We employ continuous token supervision along with L1 and L2 losses to train the model and demonstrate that each of them contributes significantly to the model.

- Sentence-Level Tokenization: We implement sentence-level tokenization, ensuring the generated results achieve the desired length despite the limitations of maximum sequence length.

- Exploring Possibilities: We further explore the potential of our method, including few-shot in-context learning and implicit modeling of shot types.

- New Metrics: We propose new ST and LT metrics to evaluate video consistency from both short-term and long-term perspectives.

We believe these contributions demonstrate the novelty, robustness, and flexibility of our approach. 

**Typos and Confusion**

We apologize for typos, grammar mistakes and unclear expressions.
We have carefully addressed the issues raised by the reviewers and made detailed revisions in the revised paper. **All revisions in the revised paper have been highlighted in red.**
Specifically, we:

- Improve Figure 2 to make it clearer and more comprehensible.
- Explicitly describe the losses introduced during DAE training for face embedding and description embedding.
- Clarify unclear expressions in the text, such as "episode's visual token."
- Correct typos, including "2kd means and 2kd variances."
- Add additional experiments to explore the importance of the L1 and L2 losses, further demonstrating that, L1 L2 together with the continuous token loss can jointly enhance model optimization.

Once again, we'd like to thank the reviewers for their careful readings and valuable comments. We welcome further discussion and are happy to make additional improvements!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel long-duration video generation framework, which is based on fine-tuning LLAVA to generate compressed tokens of key frames, which are then used to generate keyframes using a fine-tuned SDXL, and then videos using SVD. Extensive experiments demonstrate the effectiveness.

Initial reviews had mixed ratings 5686. Reviewers raised a number of concerns, including:
1. lack results of non-human videos (zniU)
2. any cherry picking done? (zniU)
3. missing evaluation on ID (xVaX)
4. boundary cases (limitations) not analyzed, e.g., will the model fail at ID consistency if one face embedding is provided (xVaX)
5. limited novelty (existing methods can do key-frame generation) (xVaX)
6. missing or unclear descriptions about method (4ibH, Jjeo) 
7. missing ablation study on objective function (4ibH)
8. training cost? (Jjeo)
9. still obvious gap for target ID evaluation (Jjeo).
10. need to use video generation metrics, not just image-level (Jjeo). 

The authors provided a response to address these concerns, including providing ablation studies, discussing limitations, and expanding on the novelty.  During discussion, authors clarified many points with 4ibH and added more interpretations of the various results / ablations. In the end, all reviewers were satisfied after the rebuttal and discussion, and final reviews were all positive 6688.  Reviewers appreciated the quality of the result, as well as the unique problem addressed.

Authors should update the paper based on the reviews, response, and discussions.