SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation

Authors: Yining Hong, Beide Liu, Maxine Wu, Yuanhao Zhai, Kai-Wei Chang, Linjie Li, Kevin Lin, Chung-Ching Lin, Jianfeng Wang, Zhengyuan Yang, Ying Nian Wu, Lijuan Wang
Primary area: generative models
Keywords: video generation, complimentary learning system, slow-fast learning, diffusion

Abstract:
Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729395858511

thread_type:
official_review

<thread_object>
creator: reviewer_bxbh

<content>
summary:
This paper proposes a slow-fast structure for retaining episodic memory in long video generation. The “slow” video model follows similarly to prior works in masked conditional video diffusion models, where x prior frames are conditioned on to generate y future frames. To incorporate longer-context memory when generating multiple chunks in a row, the authors introduce temporal LoRA parameters that are updated during inference time to store episodic memory. For more general long-context generation tasks, the paper additionally proposes a slow-fast training algorithm.

strengths:
* The paper is generally easy to follow, and well written
* The proposed algorithm is interesting, and could be a promising direction for incorporating memory through test-time training in neural networks
* Experimental results on long videos show better consistency than baseline methods, and without using their temporal LoRA module.
* The authors introduce a collection of action-conditioned video that could be useful for future related research

weaknesses:
* The purpose of the “slow” video baseline comparisons are a little confusing to me. Is this not just showing that training on specifically curated action data results in a model that can better generate conditioned on actions?
* The authors have shown some benefits in retaining long-term memory over the course of multi-chunk video generation. However, I am still a little doubtful at the long-context abilities of this algorithm if used at scale. It seems difficult to propagate / learn abstract representations in long term memory that may be useful for long-term video generation, especially since the temporal LoRA is updated using a pixel-wise diffusion loss and thus biased to memorizing a specific instance of a scene. How would this model perform in more difficult long context tasks (e.g. remembering something seen 10-30s ago, or trying to localize position / orientation in 3D space), such as the datasets introduced in [1]? Or, in a more simpler environment, making one full rotation, and then randomly turning left and right (the first rotation should newly generate the scene, and afterwards only requires accurate scene retrieval from memory). 


[1] Yan, Wilson, et al. "Temporally consistent transformers for video generation." International Conference on Machine Learning. PMLR, 2023.

questions:
* It would be useful if the authors could include some way to view the original video samples (not just snapshots of frames in the paper). Perhaps with an anonymous site?
* How long are the generated videos in seconds? Most figures show timestamps, but framerate is not mentioned in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730370037108

thread_type:
official_review

<thread_object>
creator: reviewer_ixgc

<content>
summary:
In this paper, the authors tackle the challenging action-driven video generation task. They approach it by dividing the model and training into a slow and fast learning loop, reflecting the changes in the  dynamics in the world model and in individual episodes. This is achieved by adding a temporal LoRa to the base architecture, which is also used during inference. While minimally increasing the inference time, this temporal-LoRa leads to an improved generative performance and temporal consistency.
To train the model the authors create a new large scale video-action-annotation dataset. The model performance is evaluated on video generation tasks and long-horizon planning.

strengths:
1. The paper is well structured, and additional details in the appendix help to clarify some aspects.
2. The authors introduce a bio-inspired interpretation of a slow-and fast learning loop to improve long video generation by maintining an episodic memory, and link the inspiration to the hippocampus-cortex learning hypothesis. I like the independence of the lora loop from the action label embedding and treating the previous and next chunk as a single continuum, making it easier to assess the actual impact of this architecture change.
3. A new dataset for benchmarking action-driven video generation models is introduced, leveraging existing datasets and adding new (mainly video game engine) data and labels. The authors compare their model with several s.o.t.a. models and introduce a new metric for scene consistency.
4. The model is benchmarked using several metrics, including one that requires multi-sampling with human labeling.

weaknesses:
1. The interpretation of  fast and slow memory or training loops is a staple e.g. in generative replay in the continual learning domain (including often citing the hippocampus inspiration), as well as several RNN-like architectures with slow and fast memory. A link to these previous works (even if just in the appendix) could further improve the paper.
2. Novelty: the authors cite some previous work on the LLM domain as the main source of inspiration for this usage of temp-lora. The term temp-lora and its application in the video generation domain was used in previous works such as [1] (which is only a preprint, but given it is from feb 2024 still relevant for this review), a statement how their lora application differs would be appreciated. Given these works are only preprints it does not mean they take away from the novelty of this submission. It is just to get a better understanding of the differences in how lora is applied. Even if it is the same technique it would be interesting to know if the authors think the chosen task and datasets are better suited for the technique compared to the taks & datasets in the previous works.
3. I would like to see more architectural and (hyper) parameter details of the model used in the experiments (esp. The Unet + Lora), even if it is based on existing pretrained models, and how the hyperparmeters were chosen (if different from the base models).

Other inputs (not weaknesses) that may be used to improve the paper: 
There seems to be repeating text in appendix A.2 Preliminaries On Low-Rank Adaptation

[1] Ren, Yixuan, et al. "Customize-a-video: One-shot motion customization of text-to-video diffusion models." arXiv preprint arXiv:2402.14780 (2024).

questions:
1. Is there a limit of the temp-lora in terms of the capacity of trajectories of individual objects in the scene if their motion is not the same as the global motion?
2. I did not find the number of diffusion steps T used in the experiments, please provide these numbers and any other hyperparameters used if different than the base model.
3. How is the performance of the model on dynamic objects in a scene if you re-visit an area, e.g. moving objects like cars or persons in an early episode that you re-visit later?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731102034230

thread_type:
official_review

<thread_object>
creator: reviewer_c2zj

<content>
summary:
In this paper, the authors focus on the problem of long term video generation. To this end, they propose SlowFast-VGen, a model which leverages episodic memory (using a LoRA module from long text generation) to improve context utilization over longer time horizons. The authors also argue that the approach has connections to Cognitive Science. They then evaluate their approach on multiple datasets including RLBench and Minecraft. They demonstrate convincing quantitative results that beat baselines on metrics measuring long term consistency.

strengths:
1. The paper addresses a serious failing in contemporary approaches to video modeling. Current video models cannot scale well to long time horizons, and this paper shows promise in advancing this capability.

2. The paper is well written and easy to understand.

3. Experimental evaluation is thorough, and the datasets used is reasonably diverse. The quantitative results are convincing.

weaknesses:
1. The proposed method is quite complex, requiring multiple working parts (Slow vs Fast modules).

2. I think the paper might benefit from more complex datasets. Datasets from video games, robotic simulation, and car driving are used, but these are not particularly diverse. Training/testing on a dataset such as Walking Tours (Venkataramanan et al 2024) would strengthen the paper.

questions:
1. The approach is complex. Are there any possible ways to streamline the model while still leveraging the Slow/Fast concept?

2. How would this approach deal with a complex "real world" dataset of long videos beyond just constrained domains such as Kitchens or Car Driving?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731139090342

thread_type:
official_review

<thread_object>
creator: reviewer_ifrv

<content>
summary:
The paper introduces SLOWFAST-VGEN, a dual-speed learning framework for action-driven, long video generation inspired by the complementary learning system in human cognition. This model integrates slow learning of world dynamics with fast learning for episodic memory storage, aiming to address challenges in generating consistent, coherent long videos. The slow learning component utilizes a masked conditional video diffusion model, while the fast learning component leverages LoRA to store episodic memory. The authors evaluate the model on various datasets and metrics, achieving superior FVD scores and scene consistency compared to baseline models.

strengths:
1. **Novelty in Dual-Speed Learning**: By combining slow learning of world models and fast episodic learning, SLOWFAST-VGEN represents an innovative approach, mimicking cognitive memory systems effectively.
2. **Comprehensive Benchmarking**: Evaluations include diverse tasks and metrics, reinforcing the model’s robustness across applications.

weaknesses:
1. **Limited Generalization**: SLOWFAST-VGEN’s performance may drop in scenarios outside its training domains, such as novel environments or complex, unforeseen actions, limiting its adaptability in dynamic real-world applications.
2. **Memory Inconsistency Over Long Sequences**: TEMP-LORA effectively stores episodic memory within single episodes but struggles with consistency across multi-episode or long-term tasks, potentially leading to fragmented recall in extended sequences. Since the authors mention that the slowing learning for world modeling can capture general dynamics, some long-horizon tasks need to be evaluated, such as moving to the right and then to the left.
3. **High Computational Cost**: The reliance on a diffusion model for video generation demands significant computational resources. Although using LoRA, the two-stage training. What about latent diffusion models for the slow learning, while video diffusion models for the fast learning?

questions:
1. How are the parameters optimized across the slow and fast learning phases to ensure smooth interaction without redundancy? Could alternative approaches, such as dynamically adjusting memory encoding based on the video’s complexity, further streamline the integration?
2. For tasks that require nuanced, multi-step interactions (e.g., tasks involving conditional actions or dependencies between actions), how does SLOWFAST-VGEN manage action dependencies across time steps? Are there specific mechanisms to ensure that complex sequences remain consistent and contextually appropriate?
3. What do you think about the hierarchical learning paradigm? Such as the slow/fast learning v.s. long-horizon key-frame generation + show-horizon dense-frame generation. This is just a question, not need to conduct experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732061264230

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. In addition to the response to specific reviewers, here we would like to highlight our contributions and the new experiments that we add in the rebuttal.

**[Our Contributions]**
We are glad to find out that the reviewers generally acknowledge our contributions:
* The paper is well-written and easy to follow [C2ZJ, iXgc, bXBH]
* The paper introduces a novel bio-inspired interpretation of a slow-and fast learning loop to improve long video generation by maintining an episodic memory [iFRv, C2ZJ, iXgc, bXBH]
* The authors introduce a collection of action-conditioned video that could be useful for future related research [iFRv, C2ZJ, iXgc, bXBH]
* Experimental evaluation is thorough [iFRv, C2ZJ, iXgc, bXBH]

**[New Experiments]**
In this rebuttal, we have added more supporting experiments to address reviewers’ concerns.
* Results on Action Generalization (iFRv)
* Parameter Selection based on video complexity (iFRv)
* Results on the Walking Tour dataset (C2ZJ)
* Ablation Studies on Learning Rate and Lora Rank (iXgc)
* Ablation Studies on Inference-Time Diffusion Steps (iXgc)
* Experimental Results on the Teco-Habitat Dataset (bXBH)

**[Qualitative examples]**
We include our qualitative examples of all given experiments in https://sites.google.com/view/slowfastvgen/home (Please use a computer to open the demo website. If prompted "sign up needed", you may close it and reopen after a while, or sign in to your google account, or click copy link and paste in a new browser window.)
* Examples on Action Generalization (Section 4)
* Examples on Walking Tour (Section 5)
* Examples on Scene Revisit with Dynamic Objects (Section 6)
* Action-Conditioned Long Video Generation on Teco-Habitat (Section 7)

We hope our responses below convincingly address all reviewers’ concerns. We thank all reviewers’ time and efforts again!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732558154428

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you and we are looking forward to your post-rebuttal feedback!

comment:
Dear AC and all reviewers:

Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity.

We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional experiments or clarification that we can offer to make the paper better. We appreciate your comments and advice.

Best,

Author
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation inspired by human cognitive learning systems. The paper received strong scores (6,8,8,8) and positive recommendations from all reviewers, with consistent praise for its innovative approach to addressing long-term consistency in video generation.
The reviewers highlighted several key strengths: the bio-inspired slow-fast learning framework that effectively maintains episodic memory, strong empirical results showing significant improvements over baselines (FVD score of 514 vs 782), and the creation of a valuable large-scale action-annotated video dataset. The paper was also commended for its clear presentation and thorough experimental evaluation.

Initial concerns focused on three main areas: the generalization capabilities to novel scenarios, the complexity of the approach, and questions about handling more challenging long-term dependencies. The authors provided comprehensive responses, including additional experiments on the Walking Tour dataset showing strong generalization, clarification of the model's elegant core design despite its apparent complexity, and new results on the Teco-Habitat dataset demonstrating promising capabilities for complex navigation tasks.

While some reservations remain about the model's ability to learn abstract long-term representations (as noted by reviewer bXBH), there is consensus that the paper represents an important step forward in test-time training for long video generation. The authors' rebuttal thoroughly addressed most concerns, leading one reviewer to explicitly raise their score.