Mora: Enabling Generalist Video Generation via A Multi-Agent Framework

Authors: Zhengqing Yuan, Yixin Liu, Yihan Cao, Weixiang Sun, Haolong Jia, Ruoxi Chen, Zhaoxu Li, Bin Lin, Li Yuan, Lifang He, Chi Wang, Yanfang Ye, Lichao Sun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Generation, Multi-agent, Adaption Training

Abstract:
Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI’s Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora’s functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench \cite{huang2024vbench}, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora’s 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation.

Decision: reject
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729500398614

thread_type:
official_review

<thread_object>
creator: reviewer_vz5n

<content>
summary:
This paper proposes a multi-agent framework named Mora to enable text-to-video generation by leveraging open-source models such as Llama and Stable Diffusion. The main contributions include a self-modulation factor for inter-agent coordination, a data-free training strategy and a human-in-the-loop data filtering. The authors compare the performances of Mora with Sora on six video generation tasks.

strengths:
1. The authors break down the process of video generation into several steps and utilize off-the-shelf models to implement the generation process.
2. The authors clearly describe the whole pipeline.
3. Data systhesis and filtering strategy are used to improve the quality of training.

weaknesses:
1. There are no demo videos provided by the authors, no gif files or no project website. It is hard to evaluate the quality of generated video with some sampled frames. 
2. The key component of the self-modulated training is the modulation factor. Is it able to visualized this factor?
3. How would the performances be if the agents A1 to A5 were replaced with different models?

questions:
How to explain that a multi-step generation can perform better than an end-to-end generation? For example, why a combination of open-sourced text-generation model, text-to-image model, image-to-image model and image-to-video model can outperform an end-to-end open-source text-to-video generation model and replicate a close-source model's functionalities.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730131197044

thread_type:
official_review

<thread_object>
creator: reviewer_wyhm

<content>
summary:
This paper introduces a new generalist framework named Mora that synergies Standard Operating Procedures for video generation. The proposed Mora can be used to tackle a lot  of video-related tasks. Specifically,  Mora consists of five Agents: Prompt Selection and Generation Agent, Text-to-Image Generation Agent, Image-to-Image Generation Agent, Image-to-Video Generation Agen, and Video Connection Agent. Experimental results show the effectiveness of the proposed method, and Mora can surpass the capabilities of existing leading models in several respects.

strengths:
- A new generalist Agent-based framework named Mora is proposed.
- The proposed Mora can perform a lot of video-related tasks and shows remarkable results.
- Experimental results indicate the effectiveness of the proposed method, and Mora can surpass the capabilities of existing leading models in several respects.

weaknesses:
- This paper contains a lot of existing techniques involving Prompt enhancement, Image generation, Image editing, Video generation, and Video connection. The main technical innovations should be clarified. Explaining how the system's architecture or the interaction between components differs from or improves upon existing approaches may help readers to understand.
- Since many components are involved, Mora's inference speed and computational requirements to existing state-of-the-art models for each of the video-related tasks mentioned should be discussed. Additionally, is there any optimizations implemented to improve efficiency, given the multi-component nature of the system.
- Will there be accumulated errors between multiple components? Providing empirical results showing how errors may or may not accumulate across different components and tasks is necessary. Or are there any techniques implemented to mitigate potential error accumulation between components?

---------------------------------------------------
Thanks to the authors for their responses. However, I still think that this paper involves too many existing technologies, more like a project to connect multiple technologies in series, and multiple sub-modules may have error accumulation. So I trend to maintain my score.

questions:
Please refer to Weaknesses for more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730357321933

thread_type:
official_review

<thread_object>
creator: reviewer_3ifh

<content>
summary:
The paper introduces Mora, a novel multi-agent  video generation framework, which leverages existing open-source modules to replicate Sora’s functionalities. Mora enhances inter-agent coordination through multi-agent fine-tuning with a self-modulation factor and uses a data-free training strategy to synthesize high-quality training data. It also incorporates a human-in-the-loop mechanism for data filtering. Evaluation shows Mora achieves comparable performance to Sora.

strengths:
(1) Comprehensive system: The proposed Mora is capable of handling a wide range of visual generation and editing tasks, including Text-to-Image Generation, Image-to-Image Generation, Image-to-Video Generation, and Video Connection. 

(2) Clarity of Writing: The writing is clear and easy to follow, making the paper accessible to a broader audience.

(3) Code Availability: Although the training code is not provided, the authors have shared part of their code in the supplementary materials, promoting transparency and potential reproducibility of their results.

weaknesses:
(1) Limited Novelty: The paper's primary contribution lies in utilizing a language model as an agent to call various pre-existing  visual generation models for video generation. However, neither the agent system nor the video generation models are novel contributions from the authors. Although the authors propose some additional modules to optimize the system, these present several issues, as outlined below.

(2) Concerns with Self-Modulated Fine-Tuning Algorithm: In Section 3.2, the authors introduce a self-modulated fine-tuning  algorithm that optimizes both the language model (A1) and visual generation models (A2-A5) end-to-end  using a visual generation loss. This concept is unconventional and lacks sufficient supporting research. The authors do not provide adequate citations to justify this algorithm. Additionally:

(2.1) The process is inherently difficult to optimize. The authors mention using only 96 samples for training in Section 4.2, which is unconvincing for such a complex task.

(2.2) Without releasing the training code, it is hard to assess the effectiveness of their methodology based solely on theoretical descriptions and hyperparameter choices.

(2.3) The visual results (Figures 9-12)  show significant artifacts, such as blurriness and object deformation, which raise doubts about the efficacy of the self-modulated fine-tuning algorithm.

(3) Data-Free Training Strategy and Distillation Concerns: The authors propose a data-free training strategy using large models to synthesize training data, which resembles a distillation approach. However, this strategy may cause the generated results to overfit to the large models. This calls into question the added value of using a multi-agent  approach, which typically comes with higher inference costs. The authors should provide a fair comparison to demonstrate that training Mora with this strategy yields better performance  than directly distilling a smaller model. Additionally, quantitative analysis of agent success rates and inference efficiency would strengthen the claims made about the multi-agent system’s benefits.

(4) Problem Definition in Section 3.1: The authors claim that their objective is to maximize quality metrics while ensuring diversity in the generated videos. However, quality metrics alone are not the ultimate goal for video generation. Beating existing benchmarks such as SORA does not inherently demonstrate the model's superiority. Furthermore, in an agent-based system, other factors such as module collaboration speed, accuracy, and robustness must be considered.

questions:
（1）Optimization Challenges with the Self-Modulated Fine-Tuning Algorithm:

（1.1）The self-modulated fine-tuning algorithm in Section 3.2 is unconventional and lacks supporting citations. Could the authors provide more references or evidence to justify this approach?

（1.2）Given the complexity of optimizing such a system, training with only 96 samples seems inadequate. Can the authors elaborate on how this sample size is sufficient, or provide results from larger-scale experiments?

（1.3) The visual results show significant artifacts like blurriness and object deformation. How do the authors plan to address these issues to improve visual quality?


（2）Effectiveness of Data-Free Training Strategy:

（2.1）The data-free training strategy resembles a distillation approach and may result in overfitting to large models. Can the authors show comparative experiments to demonstrate that the proposed multi-agent approach offers clear advantages over simply distilling a smaller model?

（2.2）Additionally, can the authors provide a quantitative analysis of agent success rates and inference efficiency to better justify the benefits of using the multi-agent system?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732236906947

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all reviewers for their thorough and constructive feedback which has helped strengthen our work. We have uploaded a revised manuscript with changes highlighted in red text (Appendix A.10 and Figure 7). Below we address the key questions raised among reviewers.

> Q1: Concerns about novelty and technical contributions (Reviewers `3ifH`, `WYHM`)

**Response**: While we leverage existing open-source models, our work's novelty lies in three key technical contributions. First, we introduce a novel self-modulated multi-agent fine-tuning algorithm that enhances inter-agent coordination through a self-modulation factor, allowing each agent to dynamically adjust its influence during training—an approach previously unexplored in video generation. Second, we developed a unique data-free training strategy that combines synthesized high-quality training data with human-in-the-loop mechanisms and multimodal LLMs for data filtering, effectively addressing the fundamental challenge of scarce high-quality video data. Third, our system presents a unified framework that extends to multiple video generation tasks, including text-to-video, image-to-video, video extension, editing, connection, and digital world simulation.

---

> Q2: Questions about computational efficiency and resource requirements (Reviewers `WYHM`, `VZ5N`)

**Response**: Our system is designed with efficient resource utilization in mind through sequential memory management, where each agent releases memory after completing its task, maintaining maximum memory usage at 46.98G—equivalent to the highest-requirement agent. While the primary computational cost stems from video generation, the overhead from agent coordination is negligible, as evidenced by our benchmarks: text-to-video (3s) takes 261s compared to the baseline's 243s, and image-to-video (3s) requires 253s versus the baseline's 245s. These modest increases in processing time are justified by significant quality improvements, as demonstrated in our comprehensive benchmarks across multiple tasks.

---

> Q3: Concerns about training sample size and effectiveness (Reviewer `3ifH`)

**Response**: Our approach achieves strong results with limited samples through an efficient training strategy that leverages agents' existing knowledge, focusing on improving collaboration rather than training from scratch. Through extensive experimentation with varying sample sizes from 16 to 144 samples, we discovered that performance plateaus at 96 samples (0.6482 optimal performance), with larger sample sizes showing diminishing returns (128 samples: 0.6469). The robustness of our approach is further validated through multiple out-of-domain benchmarks, including T2V-CompBench [1], EvalCrafter [2], and VBench [3], demonstrating strong generalization capabilities despite the focused training set.

---

**References**

`[1]`. Sun, Kaiyue, et al. "T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation." arXiv preprint arXiv:2407.14505 (2024).

`[2]`. Liu, Yaofang, et al. "Evalcrafter: Benchmarking and evaluating large video generation models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

`[3]`. Huang, Ziqi, et al. "Vbench: Comprehensive benchmark suite for video generative models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732611679405

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Version of Our Manuscript

comment:
We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions on our paper. During the rebuttal period, we conducted various additional experiments and incorporated the results into the revised version of our manuscript. Specifically, we have updated Sections A.10, A.11, B.3, B.4, B.5, B.6, and B.7. The modified portions are highlighted in red in the revised paper to facilitate your review. We hope our rebuttal has addressed your concerns. Please let us know if any points require further clarification before the response period concludes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733187017663

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final 12h Reminder

comment:
Dear Reviewers `3ifH` and `WYhM`,

Thank you for your valuable and thoughtful feedback throughout the review process. As the response period enters its final 12 hours, we kindly request your confirmation on whether our responses have sufficiently addressed your concerns.

We have thoroughly addressed the issues raised in your reviews, including:

1. Clarifications on the novelty of our self-modulated fine-tuning algorithm and multi-agent framework.

2. Detailed analysis of computational efficiency and inference speed.

3. Empirical results addressing potential error accumulation across components.

4. Comparisons with existing benchmarks to demonstrate Mora’s strengths.

We hope these responses have resolved any remaining doubts. However, if there are still points requiring clarification, we are fully committed to providing additional explanations promptly.

Your feedback has been invaluable in improving our work, and we sincerely appreciate your time and effort in reviewing our submission. Please feel free to share any final comments or questions at your earliest convenience before the response period concludes.

Thank you for your understanding and cooperation.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes mora, a multi-agent video generation framework based on current best existing open-source modules, to improve the generation quality. The authors propose several key modules such as inter-agent coordination and data-free training to synthesize high-quality data, as well as human-in-the-loop for data filtering, and evaluation demonstrates the effectiveness of Mora. As reviewers point out, Mora is with limited Novelty that based on the existing language model and video generation model, and also considering the computational requirements and accumulation errors issues, The AC recommend for rejection.