SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
Authors: Yining Hong
Beide Liu
Maxine Wu
Yuanhao Zhai
Kai-Wei Chang
Linjie Li
Kevin Lin
Chung-Ching Lin
Jianfeng Wang
Zhengyuan Yang
Ying Nian Wu
Lijuan Wang
Venue: ICLR 2025 Spotlight (2025)
Keywords: video generation
complimentary learning system
slow-fast learning
diffusion
Primary area: generative models

Abstract: Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well.

Published as a conference paper at ICLR 2025
SLOWFAST- VGEN: SLOW-FAST LEARNING FOR ACTION-DRIVEN LONG VIDEO GENERATION
Yining Hong', Beide Liu' Maxine Wu't, Yuanhao Zhai Kai-Wei Chang', Linjie Li², Kevin Lin?, Chung- Ching Jianfeng Wang", Zhengyuan Yang2.t, Yingnian Wu Lijuan Wang
UCLA, Mierosoft Research, State University of New York Buffalo
Co-second Contribution, Equal Advising
(a) Slow Learning World Model
(b) Fast Learning Episodic Memory
Right
t- 1-0110s forward
Loc!
Change the Left Lane
10-20s
Go left
Loc4
Pick Apple
Close Door
20-30s
right
30-400
Run Streight Ahead
Put Down the Knife
Back
Loot
(c) Memory-Augmente Video Planning
Move both cubes to the green area.
Subsequently,
original positions.
Descend to a Lower Altitude
We propose SLOWFAST- VGEN, a dual-speed action-driven video generation system that mimies the complementary learning system in human brains. The slow learning phase (a) learns an approximate world model that simulates general dynamics across diverse set of scenarios. The fast learning phase (b) stores episodic memory for consistent long video generation, e,g, generating same scene for "Locl" after traveling across different locations. Slow-fast learning also facilitates long-horizon planning tasks (c) that require the efficient storage of long-ten episodic memories.
Figure I:
ABSTRACT
Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads inconsistencies across temporally distant frames when generating longer videos, these frames fall beyond the model's context window. To this end, intro- duce SLOWFAST-VGEN a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference- time fast learning strategy based on temporal LoRA module. Specifically. the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset 200k videos with language action annotations, covering wide range of scenarios. Extensive experiments show that SLOWFAST VGEN outperforms baselines across various metrics for action-driven video generation, achieving FVD score of 514

---

Published as a conference paper at ICLR 2025
compared to 782, and maintaining consistency in longer videos, with average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm sigmificantly enhances performances on long-horizon planning tasks as well. Project Website: httpps://st /ww.://twww/ss/////////ss/sss..s www.tt-veenn.nnna.o github.ic
INTRODUCTION
Human beings are endowed with a complementary learning system (McClelland et al., 1995). The slow learning paradigm, facilitated by the neocortex, carves out a world model from encyclopedic scenarios have encountered, enabling of actions (Schwesinger, 1955). as shown in Figure !| supports fast learning, allowing for rapid adaptation to new environments and efficient storage of episodic memory ( (Tulving] 1983), ensuring that long- long-horizon simulations remain consistent (e.g., g., in Figure (b), when we travel across locations 1 to 6 and return, the scene remains unchanged.)
to make decisions by anticipating potential outcomes (a). Complementing this, the hippocampus
Recently, riding the wave of success in video generation (Blattmann et al. 2023a), researchers have introduced action-driven video generation that could generate future frames conditioned on actions (Yang et al. 2024 Wang et al. [2024a] Hu et al.) 2023 Wu et al. 2024), mimicking the slow learning paradigm for building the world model. However, these models often generate videos of constrained lengths (e.g., 16 frames for 4 secs (Xiang et al.) 2024: Wu et al. 2024)) limiting their ability to foresee far into the future. Several works focus on long video generation, using the outputs of one step as inputs to the next (Harvey et al., [022]] Villegas et al., 2022 Chen et al. 2023) Guo et al.) 2023a Henschel et al., 2024). Due to the lack of fast learning, these models often do not memorize trajectories prior to the current context window, leading to inconsistencies in long-term videos.
In this paper; we propose SLOWFAST- which seamlessly integrates slow and fast learning in a dual-speed learning system for action-driven long video generation. For the slow learning process, we introduce masked conditional video diffusion model that conditions on language inputs denoting actions and the preceding video chunk, to generate the subsequent chunk. In the inference time, long video generation can be achieved in a stream-in fashion, where we consecutively input actions for the current context window, and the succeeding chunk can be generated. To enable the model to memorize beyond the previous chunk, we further propose a fast learning strategy, which rapidly adapts to new contexts and stores episodic memory, enabling the model to maintain coherence over extended sequences. By combining slow learning and fast learning,
FAST- VGEN can generate consistent, action resiion-espposvvee long videos, as shown in Figure [[ (a) and (b).
A key challenge for fast learning in long video generation resides in maintaining consisteney across long sequences, while avoiding computationally intensive or -nnffrr-neffficientt strategies for stor- ing lengthy video sequences. To address this, we propose novel inference-time fast learning strategy that stores episodic memory in Low-Rank Adaptation (LoRA) parameters. Specifically, we draw inspiration from Temporary LoRA (Wang et al.) 2024b), originally developed for long text generation, and extend this concept for fast learning in video generation. In its original form, TEMP-LORA embeds contextual information in temporary LoRA module for language model- ing by progressively generating new text chunks based on inputs and training on these onput-output pairs. Our approach modifies the update mechanism of TEMP-LORA parameters by discarding the input-to-output format. At each context window, our TEMP-LORA for fast learning operates as follows: we generate a new video chunk conditioned on both the input chunk and the corresponding action, then concatenate the input and output into a seamless temporal continuum. This sequence undergoes a noise injection process, creating temporally coherent learning signal. We then train the denoising UNet on this noise- augmented, action-agnostic sequence. This method emphasizes mem- orizing entire trajectories rather than focusing on immediate input-output streams or transitions. By leveraging the forward diffusion and reverse denoising processes over the concatenated sequence, we effectively consolidate sequential episodic memory in the TEMP-LORA A parameters of the UNet, enabling it to maintain coherence across extended video sequences. Qualitative examples in Figure 4 and supplementary materials demonstrate that fast learning enhances the consistency, smoothness, and quality of long videos, as well as enabling the fast adaptation to test scenes.
While our fast learning module excels at rapidly adapting to new contexts and storing episodic memory, there exist specific context-aware tasks that require learning from all previous long- term episodes rather than just memorizing individual ones. For instance, in the planning section of Figure (c). the task requires not only recalling the long-term trajectory within each episode, but also
2

---

Published as a conference paper at ICLR 2025
leveraging prior experiences across multiple episodes to develop the general skill of returning the cubes. Therefore, we introduce the slow-fast learning loop by integrating TEMP-LORA into the slow learning process. The inner fast learning loop uses TEMP-LORA to swiftly adapt to each episode, preparing a dataset of inputs, outputs, and corresponding TEMP-LOR, parameters for the slow learning process. The outer slow learning loop then leverages the multi-episode data to update the model's core weights with frozen TEMP-LORA parameters.
To further boost slow learning and enable action-driven generation with massive data about world knowledge, we collect a large-scale video dataset comprising 200k videos paired with language ac- tion annotations. This dataset covers myriad of tasks, including Games, Unreal Simulations, Hu- man Activities, Driving, and Robotics. Experiments show that our FLOWFAST- VGEN outperforms baseline models on series of metrics with regard to both slow learning and fast learning, including FVD, PSNR, SSIM, LPIPS, Scene Cuts, and our newly-propssed Scene Return Consistency (SRC). Furthermore, results on two aarffuly-desiinne long- long-horizon video planning tasks demonstrate that the slow-fast learning loop enables both efficient episodic memory storage and skill learning.
To sum up, our contributions are listed as below:
We introduce SLOWFAST- VGEN, a dual-speed learning system for action-driven long video generation, which incorporates episodic memory from fast learning into an approximate world model, enabling consistent and coherent action-responsive long video generation.
We develop an inference-time fast learning strategy that enables the storage of long-term episodic memory within LoRA parameters for long video generation. We also introduce a slow-fas wow-fast learn- ing loop that integrates the fast learning module into the slow learning process for context-aware skill learning on multi-episode data.
We develop a masked conditional video diffusion model for slow learning and create a large-scale dataset with 200k videos paired with language action annotations, covering diverse categories. Experiments show that SLOWFASTE VGEN outperforms various baselines, achieving an FVD score of 514 versus 782, and averaging 0.37 scene cuts compared with 0.89. Qualitative results highlight the enhanced consistency, smoothness and quality in generated long videos. Addition- ally, results on long-horizon planning tasks prove that the slow-fast learning loop can effectively store episodic memory for skill learning.
2 RELATED WORKS
Text-to- Video Generation Nowadays, research on text-conditioned video generation has been
evolving fast (Ho et al.
2022b al Singer et al.| 2022 Luo et al.] [023 Esser et al. 2023] Blattmann
et al. 2023b Zhou et al. 2023 Khachatryan et al. [023] Wang et al. 2023), enabling the generation
of short videos based on text inputs. Recently, several studies have introduced world models capable
of generating videos conditioned on free-text actions (Wang et al. 2024a| Xiang et al. 2024 Yang
(ct al. 2024). While these papers claim that a general world model is pretrained, they often struggle to simulate dynamics beyond the context window, thus unable to generate consistent long videos. There are also several works that formulate robot planning as a text-to-video generation problem (Du et al. 2023 Ko et al.] 2023). Specifically, [uu al. (2023) trains video diffusion model to predict future frames and gets actions with inverse dynamic policy, which is followed by this paper.
Long Video Generation A primary challenge in long video generation lies in the limited number of frames that can be processed simultaneously due to memory constraints. Existing techniques often condition on the last chunk to generate the next chunk et al.) 2022; Villegas et al. 2022
Chen et al. 2023: Guo et al. 2023a Henschel et al., 2024| Zeng et al. 2023 Ren et al. 2024). which hinders the model's ability to retain information beyond the most recent chunk, leading to inconsistencies when revisiting earlier scenes. Some approaches use an anchor frame (Yang et al., 2023 Henschel et al. 2024) to capture global context, but this is often insufficient for memorizing the whole trajectory. Other methods generate key frames and interpolate between them (He et al.) 2023 Ge et al. [002 Harvey et al. 2022] Yin et al. 2023), diverging from how world models simu- late future states through sequential actions, limiting their suitability for real-time action streaming, such as in gaming contexts. Additionally, these methods require long training videos, which are hard to obtain due frequent shot changes in online content. In this paper, we propose storing episodic memory of the entire generated sequence in LoRA parameters during inference- fast learning, enabling rapid adaptation to new scenes while retaining consistency with limited set of parameters.
Low-Rank Adaptation LoRA (Hu et al. 202T) addresses the computational challenges of fine- tuning large pretrained language models by using low-rank matrices to approximate weight changes,
3

---

Published as a conference paper at ICLR 2025
reducing parameter training and hardware demands. It enables efficient task-switching and personal- ization without additional inference costs, as the trainable matrices integrate seamlessly with frozen weights. In this work, develop TEMP-LORA module, which adapts quickly to new scenes during inference and stores long-context memory in its parameters.
3 SLOWFAST- VGEN
In this section, we introduce our SLOWFAST-- -OEEANTTVGEN framework. We first present the masked dif- fusion model for slow learning and the dataset collected (Sec Slow learning enables the generation of new chunks based on previous ones and input actions, while lacking memory reten- tion beyond the most recent chunk. To address this, introduce fast learning strategy, which can store episodic memory in LoRA parameters (Sec 3.2 Moving forward, we propose a slow-fast learning loop algorithm (Sec 3.3) that integrates TEMP-LORA into the slow learning process for context-aware tasks that require knowledge from multiple episodes, such as bong-horizon planning. We also illustrate how to apply this framework for video planning (Sec (.2) and conduct thorough investigation and comparison between our model and complementary learning system in cognitive science, highlighting parallels between artificial and biological learning mechanisms (See 3.5).
Slow Learning: Pretraining on All Data Fast Learning: Storing Episodic Memory in TEMP-LORA
Conditioned
Chunk
Video Chunk
Diffusion
TEMP- LORA
Action
Noise
Input Chunk!
Generate
2
Denolsing Unet + 0
Denoising Unet
TEMP- B
Action: Back off Generated
Ground-Trun Truth Output
Action
Generate
Chunk
Inference- Time Training
Action
Prompt:
Back Off
Train
LoRA
TEMP-LORA
Iteration i
Generaaee
Learning Loop: Provide
learning ->
Outar Slow Learning Loop: Provide frozen & for leaming
Figure 2: SLOWFAST- VGEn Architecture. The left side illustrates the slow learning process, pretraining on all data with a masked conditional video diffusion model. The right side depicts the fast learning process, where TEMP-LORA stores episodic memory during inference. Stream- in actions guide the generation of video chunks, with TEMP-LORA parameters updated after each chunk. In our slow-fast learning loop algorithm, the inner loop performs fast learning, supplying TEMP-LORA parameters from multiple episodes to the slow learning process, which updates slow learning parameters & based on frozen fast learning parameters.
3.1
SLOW LEARNING
3.
MASKED CONDITIONAL VIDEO DIFFUSION
We develop slow learning model based on the pre-trained ModelScope model (Wang al. 2023). which generates videos from text prompts using a latent video diffusion approach. It en- codes training videos into latent space gradually adding Gaussian noise to the latent via = Va,za + vI dhe. A Unet architecture, augmented with spatial-temporal blocks, is re- sponsible for denoising. Text prompts are encoded by the CLIP encoder (Radford al., 2021).
To better condition on the previous chunk, we follow Voleti et al. (002) for masked conditional video diffusion, conditioning on past frames in the last video chunk generate frames for the subsequent chunk, while applying masks on past frames for loss calculation. Given fy past frames and fg frames be generated, we revise the Gaussian diffusion process to:
2t. = 20,5
(.ff)))ff / V0rzo.ff + VI ārE
st.is, 2t.fp((f+(())
where 2t,,, is the latent the first fp frames at diffusion step f, which is clean as we do not add noise the conditional frames. 2.ff:(fatf.) corresponds the latent of the ground-truth output frames, which we add noise to for conditional generation of the diffusion process. We concatenate (0) them together and send them all into the UNet. We then get the noise predictions out of the
4

---

Published as a conference paper at ICLR 2025
UNet, and apply masks to the losses corresponding the first fp frames. Thus, only the last fg frames are used for calculating loss. The final loss is:
-0. e...0))) [[]e- co (Z (fp + fg)l. e) [|
0.7em
Here, represents the conditioning (e.g., text), and o denotes the UNet parameters. Our model is able to handle videos of arbitrary lengths smaller than the context window size (i.e., arbitrary fp and fg). For the first frame image input, fp equals 1.
3.1.2 DATASET COLLECTION
Our slow learning dataset consists of 200k data, with each data point in the format of (input video chunk, input free-text action, output video chunk). The dataset can be categorized into 4 domains:
Unreal. We utilize the Unreal Game Engine (Epic Games) for data collection, incorporating environments such as Google 3D Tiles, Unreal City Sample, and various assets purchased online. We introduce different agent types (e.g.. human agents and droids) and use a Python script to automate action control. We record videos from both first-person and third third-person perspectives, capturing keyboard and mouse inputs as actions and translating them into text (e.g., "go left"). Game. We manually play Minecraft, recording keyboard and mouse inputs and capturing videos. Human Activities. We include the EPIC-KITCHEN (Damen et al.) 2018 2022) dataset, which comprises extensive first-person (egocentric) vision recordings of daily activities in the kitchen. Robot. We use several datasets from Deeeo-mmmmneent (Collaboration et al. 2024), well as tasks from Metaworld (Yu et al. 2021) and RLBench (James et al., 2019). As most robot datasets consist of short episodes (where each episode is linked with one language instruction) rather than long videos with sequential language inputs, we set Sp to for these datasets.
Driving. We utilize the HRI Driving Dataset (HDD) (Ramanishka et al., (2018), which includes 104 hours of real human driving. We also include driving videos generated in the Unreal Engine.
3.2 FAST LEARNING
The slow learning process enables the generation of new video chunks based on action descrip- tions. A complete episode can thus be generated by sequentially conditioning on previous outputs. However, this method does not ensure the retention of memory beyond the most recent chunk, po- tentially leading to inconsistencies among temporally distant segments. In this section, we introduce the novel fast learning strategy, which can store episodic memory of all generated chunks. We begin by briefly outlining the generation process of our video diffusion model. Subsequently, we introduce temporary LoRA module, TEMP-LORA A, for storing episodic memory.
1). We initialize new chunks with random noise at 24. During each step, we combine the previous iteration's 's clean output =j with the current noisy latent zς. This combined input is fed into a UNet to predict and remove noise, producing less noisy version for the next step. We focus only on the newly generated frames, masking out previous-iteraiie
Generation Each iteration consists of T denoising steps (t = T .)..1).
outputs. This process repeats until the iteration's 's final step, yielding clean latent z). The resulting z becomes input for the next iteration. The generation process works directly with the latent representation, progressively extending the video without decoding and re-encoding from pixel space between iterations.
TEMP-LORA Inspired by Wang et al. (002bb), we utilize a temporary LoRA module that embeds the episodic memory in its parameters. T TOMA-LORA A was initially designed for long text generation, progressively generating new text chunks based on inputs, and use the generated chunk as ground- truth to train the model conditioned on the input chunk. We improve TEMP-LORA for video gen- cration to focus on memorizing entire trajectories rather than focusing on immediate input-output streams. Specifically, after the generation process of iteration i, we use the concatenation of the input latent and output latent at this iteration to update the TEMP-LORA parameters.
2 2
20:
2 Vārza + v1 O4f6
(3)
L(O, () E,,e'' (0(11)) [||- co. (ff. 0013
Specifically, we concatenate the clean output from the last iteration =5 (which also the input of the current iteration) and the clean output from the current iteration =1 to construct a temporal continuum z6'. Then, we add noise the whole =f' sequence, which results in 2f' noise diffusion step Note that here we do not condition on clean 2 anymore, and exclude text conditioning to
5

---

Published as a conference paper at ICLR 2025
Algorithm SLOWFAST-VGBN Algorithm
(a) Fast Learning
Input: First frame of video sequence Xo. total generating iterations I, video diffusion model with
frozen pretrained weights & and LoRA parameters Ou fast learning learning
Output: Long Video Sequence y
1: Ao VAE_ENCODE(2))
(Encode into latents)
2: for i in to to do
3:
// Generate the sequence in the current context window
4: if + then --1-1
6:
end if
7:
C. User Input())
[Action conditioning acquired through user interface input)
8:
V: (& +..,,, C.)
V
|Concatenate the output latents the final sequence latents)
10:
// Use the input and output in the context window to train TEMP-LORA (Concatenate input and output to prepare TEMP-LORA training data)
11:
X; = X; yi
12: Sample Noise V on the whole At sequence
13: Calculate Loss the whole X; sequence
O.41 t O a VoLoss
15: end for
16: y = VAE_DECODEN)
(Decode into video}
17: return y
(b) Slow-Fast Learning Loop
Definition: lask-specific slow learning weights 4, lask-specific dataset D, LoRA parameters of all episodes e, slow learning learning rate 8, slow-learning dataset D,
// Slow Learning Loop
I: while not converged do
2: Do + //prepare dataset for slow learning
3: for each sample (zr, episode) in 1 do
4: // Fast Learning Loop
5: Suppose episode could be divided into I short sequences: A, for in to I -
Initialize TEMP-LORS parameters for episode Oj
7: for i in to I do
8:
D. D. U ( θ
(Xf+1 is the ground-truth output of input X;}
9:
Fix & and update O using fast leamning algorithm
10:
end for end for
11:
12:
Use the D, dataset for slow learning update
13:
for [X X..1 in D, do
14:
Φ. :== +
Caleulate Loss based on model output input X, ground -ruth output X
16:
Fix O and update only: & +- d- - B. Lososs
17:
end for
18:
end while
focus on remembering full trajectories rather than conditions. Then, we update the TEMP-LORA parameters of the UNet θ using the concatenated noise-augmented, action-agnostic zj' Through forward diffusion and reverse denoising, effectively consolidate sequential episodic memory in the TEMP-LORA parameters. The fast learning algorithm is illustrated in Algorithm (a).
3.3 SLOW-FAST LEARNING LOOP WITH TEMP-LORA
Previously, we develop TEMP-LORA for inference-time training, allowing for rapid adaptation to new contexts and the storage of per-episode memory in the LoRA parameters. However, some specific context-aware tasks require learning from all collected long-term episodes rather than just memorizing individual ones. For instance, solving a maze involves not only recalling the long- term trajectory within each maze, but also leveraging prior experiences to generalize the ability to navigate and solve different mazes effectively. Therefore, enhance our framework's ability to solve various tasks that require learning over long-term episodes, we propose the slow-fast learning loop algorithm, which integrates the TEMP-LORA modules into dual-speed learning process.
We detail our slow-fast learning loop algorithm in Algorithm T (b). Our slow-fast learning loop consists of two primary loops: inner loop for fast learning and outer loop for slow learning.

---

Published as a conference paper at ICLR 2025
The inner loop, representing the fast learning component, utilizes TEMP-LORA for quick adaptation to each episode. It inherits the fast learning algorithm in Algorithm I (a) and updates the memory in the TEMP-LORA throughout the long video generation process. Crucially, the inner loop not only generates long video outputs and updates the TEMP-LORA A parameters, but also prepares training data for the slow learning process. It aggregates inputs, ground-truth outputs, and corresponding TEMP-LORA parameters in individual episodes into a dataset D.. The outer loop implements the slow learning process. It leverages the learned frozen TEMP-LORA parameters in different episodes to capture the long-term information of the input data. Specifically, it utilizes the data collected from multiple episodes by the inner loop. which consists of the inputs and ground-truie outputs of each iteration in the episodes, together with the memory saved in TEMP-LORA up to this iteration.
While the slow-fast learning loop may be intensive for full pre-training of the approximate world model, it can be effectively used in finetuning for specific domains or tasks requiring long-horizon planning or experience consolidation. A key application in robot planning, where slow accumu- lation of task-solving strategies across environments can enhance the robot's 's ability to generalize to complex, multi-step challenges in new settings, as illustrated in our experiment (Sec
3.4 VIDEO PLANNING
We follow Du et al. (2023), formulating task planning lext-conditioned video generation prob- lem using Unified Predictive Decision Process (UPDP). We define a UPDP as G = (X, C, H, ø). where X is the space of image observations, C' denotes textual task descriptions, H E T is finite horizon length, and <(-[z0.c) X x C A(X is our proposed conditional video gener- ator. d synthesizes a video sequence given an initial observation Zi and text description c. To transform synthesized videos into executable actions, we use a trajectory-task conditioned policy I (1|xr17-0 c) XT x - ((AT-1). Decision-making is reduced to learning 6, which gener- ates future image states based on language instructions. Action execution translates the synthesized video plan [f]' to actions. We employ an inverse dynamics model to infer necessary actions for realizing the video plan. This policy determines appropriate actions by taking two consecutive image observations from the synthesized video. For long-horizon planning, ChatGPT decomposes tasks into subgoals, each realized as a UPDP process. After generating a video chunk for a subgoal, we use the inverse dynamics model to execute actions. We sequentially generate video chunks for subgoals, updating TEMP-LORA throughout the long-horizon video generation process.
3.5 RETHINKING SLOW-FFST LEARNING IN THE CONTEXT OF COMPLEMENTARY LEARNING SYSTEMS
Slow-Fast Learning as in Brain Structures
In neuroscience, the neocortex is associated with slow learning, while the hippocampus facilitates fast learning and memory formation, thus forming a complementary learning system where the two learning mechanisms complement each other. While slow learning involves gradual knowledge acquisition, fast learning enables rapid formation of new memories from single experiences for quick adaptation to new situations through one-shot contextual learning. However, current pre-training paradigms (e.g., LLMs or diffusion models) primarily emulate slow learning, akin to procedural memory in cognitive science. In our setting, TEMP-LORA serves as analogy to the hippocampus.
TEMP-LORA as Local Learning Rule
It's long believed that fast learning is achieved by local learning rule (Palm given pairs of patterns (x* y") to be stored in the matrix C, the process of storage could be formu- lated by the following equation:
2013). Specifically,
Σ x y
Consider a sequential memory storage process where learning steps involve adding ouputt pairs (x"", y") to memory, the change Ac₁; in each memory entry depends only on local oututt interactions Palm 2013):
Ac(t) = x (t) (t)
(5)
This local learning rule bears a striking resemblance to LoRA's update mechanism.
W' = W + AW W stow + W Fast = + + e
(6)
7

---

Published as a conference paper at ICLR 2025
Where Wist is achieved by the matrix change, updated based on the current-iteration input and output locally as in Equation B (AW + 20,1-1 + 20,5).
Slow-Fast Learning Loop Computational Analogue to Hippocampus-Neocortes Interplay
The relationship between TEMP-LORA and slow learning weights mirrors the interplay between hippocampus and neocortex in complementary learning systems. This involves rapid encoding of new experiences by the hippocampus, followed by gradual integration into neocortical networks (McClelland et al. 1995). As in Klinzing et al. (2019), memory consolidation is the process where hippocampal memories are abstracted and integrated into neocortical structures, forming general knowledge via offline phases, particularly during sleep. This bidirectional interaction allows for both quick adaptation and long-term retention (Kumaran et al., 2016). Our slow-fast learning loop fast- Here, Wtast (TEMP-LORA A) rapidly
emulates this process, where W W + AW = Wstow + Wfast- adapts to new experiences, analogous to hippocampal encoding, while slow gradually incorporates this information, mirroring neocortical consolidation.
4 EXPERIMENT
For our experimental evaluation, we will focus on assessing the capabilities of our proposed ap- proach with regard two perspectives: video generation and video planning. We will detail our baseline models, datasets, evaluation metrics, results, and qualitative examples for each component. Please refer to the supplementary material for experimental setup, implementation details, human evaluation details, computational costs, more ablative studies and more qualitative examples.
4.1 EVALUATION ON VIDEO GENERATION
Baseline Models and Evaluation Metrics We compare our FAST VGEN with several base- lines. AVDC (Ko et al. 2023) is a video generation model that uses action descriptions for training video policies in image space. Streaming- T2V (Henschel et al., 2024) is a state-of-the of-the-art text-to- long-video generation model featuring a conditional attention module and video enhancer. We also evaluate Runway (Runway) in an off-the-shelf manner to assess commercial rext-to-video generation models. Additionally, AnimateDiff (Guo et al.) 2023b) animates personalized T2I models. SEINE (Cen et al. 2023) is short-to-lone video generation method that generates transitions based text descriptions. iVidcoGPT (Wu et al., (024) is an interactive autoregressive transformer framework. We tune these models (except for Runway) on our proposed dataset for fair comparison.
We first evaluate the slow learning process. which takes in a previous video together with an input ac- tion, and outputs the new video chunk. We include series of evaluation metrics, including: Fréchet Video Distance (FVD), Peak Signal-to-Noise Ratio (PSNR). Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS). We reserved portion of our collected dataset as the test set, ensuring no scene overlap with the training set. We evaluate the model's ability to generate consistent long videos through an iterative process. Starting with an initial image and ac- tion sequence, the model sequentially generates video chunks by conditioning on the previous chunk and the next action. To assess the consistency of the generated sequence, we use Short-term Content Consistency (SCuts))Hensehee frames. We utilize DeSceneDetect (PySecneDetect) to detect scene cuts and report their number. We also introduce Scene Revisit Consistency (SRC), new metric that quantifies how consistently a video represents the same location when revisited via reverse actions (e.g., moving forward and then back). SRC is computed by measuring the cosine similarities between visual features (using ResNet-18 18 (He et al. 2015)) of the first visit and subsequent visits. We construct an SRC bench- mark in Minecraft and Unreal, automating script to generate specific navigation paths that involve revisiting the same locations multiple times during the course of a long video sequence. Finally, we conduct Human Evaluation, where human raters assess video quality, coherence, and adherence to actions, with each sample rated by at least three individuals on scale from 0 to 1.
et al. (2024), which measures temporal coherence between adjacent
Result Analysis In Table 4 we include the experimental results of video generation, where the left part focuses on slow learning for action-conditioned video generation, and the right part eval- uates the ability to generate consistent long videos. From the table, we can see that our model outperforms others with regard to both slow learning and long video generation. Specifically, our method achieves a notable FVD score of 514, significantly lower than those of other methods, such as AVDC (1408) and Runway Gen-3 Turbo (1763), demonstrating its capability to capture the un- derlying dynamics of video content. The high PSNR and SSIM scores also indicate that our model achieves high visual fidelity and similarity with ground-truth videos. Runway produces less satis-
8

---

Published as a conference paper at ICLR 2025
FVD
PSNR 1
SSIM
LPIPS
SCuts
SRC 1
Human
AVDC
1408
16,96
52.63
20.65
3.13
83.89
0.478
Streaming-T22 T2V
990
14.87
48.33
33.00
0.89
91.02
0.814
Runway Gen-3 Turbo
1763
11.15
47.29
52.71
2,46
80.26
0.205
AnimateDiff
782
17.89
52.34
33.41
2.94
90.12
0,872
SEINE
919
18.04
54.15
35.72
1.03
88,95
0.843
iVidcoGPT
1303
13.08
31.37
27.22
1.32
82.19
0.536
Ours (wo/ Temp-LoRA)
1.88
89.04
0.869
Ours LLOWFAST-VGEN
514
19.21
60.53
25.06
0.37
93.71
0.897
Table 1: Video Generation Results. FLOWFAST- VGEN outperforms baselines both in slow learn- ing and long video generation, while achieving good scores in human evaluation.
factory results and could not comprehend text inputs, suggesting that state-of-the-art commercial models still struggle to serve as action-conditioned world models. From the long video generation perspective, we can see that our model and Streaming- T2V outperform other models by a large mar- gin, with fewer scene changes (indicated by SCuts) and better scene revisit consistency. We attribute the notable improvement in performance to the mechanisms of the two models specially designed still inferior our model,
to handle short- and long-term memory. However, Streaming-T2V since the appearance preservation module it uses only takes in one anchor frame, neglecting the full episode memory along the generation process. On the other hand, our TEMP-LORA module takes into account the full trajectory, therefore achieving better consistency. We also observe that AVDC suffers from incoherence and ambiguity in output videos, which might be blamed to the image-space diffusion, which results in instability and low quality compared with latent-space diffusion models.
Qualitative Study Figure 3 shows the results of slow learning. Our model is able to encompass a variety of scenarios action- conditioond video generation, including driving scenes, robotics, human avatars, droid scenes etc. The model is able to condition on and conform to various actions (e.g., in the third row, manages to go in different directions given different prompts). We also observe that our model adheres to physical constraints. For instance, in the last action for row three, knows to come to a halt and bring the hands to resting position. In Figure 4 model's 's results with and without the TEMP-LORA module. The TEMP-LORA improves quality and consistency long video generation, demonstrating more smoothness across frames. In the first example, the scene generated without TEMP-LORA begins to morph t 96, and becomes pletely distorted and irrelevant at = 864. The scene generated with TEMP-LORA almost remains unchanged for the first 400 frames, exhibiting only minor inconsistencies by 896. Our model is able to generate up to 1000 frames without significant distortion and degradation. Additionally, the model lacking TEMP-LORA suffers from severe hallucinations for the second example, such as generating an extra bread at = 48, changing the apple's 's color at t = 144, and adding a yellow cup at 168. Although the robotics data does not involve long-term videos with sequential instructions, but rather short episodes of single executions, our model still performs well by conditioning on pre- vious frames. With TEMP-LORA, our model also produces less noise in later frames (e.g.. ., 216). As the scene is unseen in training, our model also demonstrates strong adaptation ability.
we compare our
Change to the Right Lane
Turn on Faucel
Reverse
Take the Knife
Tumn right at the intersection
Push the Button
Initial
Go Forward
Back Off
Go Right
Figure Qualitative Examples of Video Generation with Regard to Slow Learning. Our SLOWFAST- VGEN is able to conduct action-conditionee video generation in diverse scenarios. 4.2 EVALUATION ON LONG-HORIZON PLANNING
In this section, we show how SLOWFAST- VGEN could benefit long-horizon planning. We carefully design two tasks which emphasize the memorization of previous trajectories, in the domains of robot manipulation and game navigation respectively. We follow the steps in Section 4.2 for task execution and employ the slow-fast learning loop for these two specific tasks separately. We report the distance to pre-defined waypoints well the FVD of the generated long videos.

---

Published as a conference paper at ICLR 2025
1 0-32 Forward
1= 96-128 Forward
t= 320-362
t= 352-3844
1= 864-896
-806-9228
right
Wol Teipp- LoieA
Temp-la LOR
0-24
== 24-48
== 48-72
1 72-96
t= 96-120
1= 120 144
1 144-168
1= 168-182 Move the
192-216 Put the
1 216-240
Plak
Put Pick
Place the
Pick the
Place square Pick up the apple
Move the gray the oven
green tup
cup square
square again
apple lowards the gray bowl bow!
the gray
white bowl
white howl
the table
Temp-LoRA
Temp-LoRA
Figure 4: Qualitative Examples of Fast Learning for Video Generation. means frame number.
Red boxes depotpbbjects incogsistenhgyitthbeee. WeMEEnARR hopsonsoowi irplag videos.
Dist
FVD
Dist
FVD
Dist
FVD
Dist
FVD
Dist
FVD
RLBench
0.078
81.4
0.022
68.2
0.080
70.3
0.055
69.8
0.013
65.9
Minecraft
5.57
526
2.18
497
6.31
513
2.23
501
1.51
446
Table 2: Long-horizon Planning Results. Our model outperforms baselines in both tasks.
Robot Manipulation We build a new task from scratch in the RLBench (James et al. 2019) envi- ronment. The task focuses moving objects and then returning them previous locations. We record the distance to the previous locations of the cubes. Table 4.2 shows that our model and Streaming-T2V outperform models without memories. Streaming T2V has satisfying results since the appearance preservation module takes an anchor image as input, yet is still inferior to our model. Ablative results of "Ours wo Loop" also demonstrate the importance of the slow-fast learning loop.
to the pool
Go the tree
Proceed to the pool, then navigate the tree. Subsequenty, return the origin by reteacing initial path.
Go buck pool via the path
Go back to urigin
Move the black cube the groen
Move pink cube to the
Move black cube to the
Move pink cabe the
Move the pink cube black cube's location
green area. Subsequently. them their original positions. Then, put the second culte where the first the green
green
original original location
Figure 5: Qualitative Example of Planning. SLOWFAST- VGEN can retain long-term memory.
Game Navigation We develop a task in Minecraft that requires the gamer to retrace the initial path to return point. We define set of waypoints along the way, and measure the closest distance to these waypoints. From Table 4.2 and Figure we can see that our model achieves superior results.
5 CONCLUSION AND LIMITATIONS
In this paper, we present SLOOWFAST- VGEN, dual-speed learning system for action-driven long video generation. Our approach combines slow learning for building comprehensive world knowl- edge with fast learning for rapid adaptation and maintaining consistency in extended videos. Through experiments, we demonstrate SLOWFAST: VGEN' 's capability to generate coherent, action- responsive videos and perform long-horizon planning. One limitation is that our model could not suffice as an almighty world model as many scenarios are not covered in the dataset. Our method also introduces additional computation during inference specified in the Supplementary Material.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The work was done when Yining Hong was an intern at Microsoft Research. The work was also partially supported by NSF DMS-2015577, NSF DMS- NNS-224,522,, and a gift fund from Amazon. We would like to thank Yan Wang and Dongyang Ma for insights into TEMP-LORA for text generation. We would also like to thank Siyuan Zhou, Xiaowen Qiu, Xingcheng Yao and Johnson Lin for in- sights into video diffusion models. We would like to thank PLUSLab paper clinic for proofreading.
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, and et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023b. URL https::t/tpt.iiieeeeeee http://awww...or//////////./// 08818
Xinyuan Chen, Yaohui Wang, Lingjun Zhang. Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu, Seine: Short-to-long video diffusion model for generative transition and prediction, 2023. pRtps:::://axxit..
Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Man- dlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai. Anchit Gupta, Andrew Wang. Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ccola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang. Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang. Huy Ha, Igor Mordatch, Ilija Ra- dosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo. Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan. Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yun- liang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo. Mohan Kumar Srirama, Mohit Sharma. Moo Jin Kim. Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Nor- man Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart'in-Marr in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore. Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany,
11

---

Published as a conference paper at ICLR 2025
Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vin- cent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yuah-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang. Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024. URL pettps://arxiv..rrggg
02.331008800 088844
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scal- ing egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100. Interna- lional Journal of Computer Vision (IJCV), 130:33-55, 2022. URL httpt:///d// http///iioogg/....
1007/ 1263-2633-22 -1531-1
Yilun Du, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Joshua Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023.
URL https::/aaxxi..../. org/ hwt.gg/ab//32....o 00111
Epic Games. Unreal engine.
com
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content content-guided guided video synthesis with diffusion models, 2023. URL
wwt.oo/g//aooo. wtt.oorg/a/b/////o/// /ttpss///eaaa..orrr////22.2222. 03011
songwei Ge. Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with Line-agnosti vggan and time-sensitive transformer, 2022. (tRthhtts://arxtttt wrw.worrgggoooo.... abs 2204. 03668
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models, 2023a. tts/ttts://arxiv.. abs/ / 2311 16933
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. 2023b.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling long videos, 2022. URL
11 11495
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015, UKUntppss///xxxv..rrr/////////s 03385
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity long video generation, 2023. URL httpt:://axii...ot wbw.///////a//..eeoooo//eee abs//21s/2211 13221
Roberto Henschel. Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text, 2024. URL https: //aaxivvv... orgg abs/
2403. 14773
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole. Mohammad Norouzi, J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022a. URL
2210. 02303
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022b. URL wttptttss//.ii..ooooto org/ 03458
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. 2023.
URL wttphhtsss//aaiiioooooee http://wwwi...rr//////..... 17000 17080
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
7/ //rxxiv.org
09685
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark and learning environment, 2019. URL https://arxiv.orr abs/
1909. 12271
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators, 2023. URL
J. G. Klinzing, N. Niethard, and Born. Mechanisms of systems memory consolidation during sleep. Nature Neuroscience, 22:1598-1610, October 2019. doi: 10.1038:441533-009-46655. Received 18 February 2019; Accepted 12 July 2019; Published 26 August 2019.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023, URL https://axxiv.. /
abs/ / 2310.08877 08576
Dharshan Kumaran, Demis Hassabis, and James L.. McClelland. What learning systems do intelli- gent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512-534, July 2016. doi: 10. 10.11016/..tiss...oo 201.............00
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation, 2023. URL
abs 08320
James L. McClelland, Bruce L. McNaughton, and Randall C. O' O'Reilly. Why there are complemen- tary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419-477, July 1995. doi: 10.1037/0033-2955 295........1100
Günther Palm. Neural associative memories and sparse coding. Neural Networks, 37:165-171, 2013.
PySceneDetect. Pyscenedetect. URL / com/ 03-03.
Accessed: 2024-
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Gob, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene under- standing: dataset for learning driver behavior and causal reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation, 2024. URL https:
13

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutionaa networks for biomedi- cal image segmentation, 2015. URL http://aaaa....rrr//a/5555.....0 04597
Runway. Runway. URL
com/ Accessed: 2024-03-03.
Gladys C. Schwesinger. Review of "psychological development" by norman I. munn. Pedagogical Seminary and Journal of Genetic Psychology, 55xxxx-XxX, 1955.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video vixeooo-video generation without text-video data, 2022. URL https: /stpss//arxiv.. wwg.oorggaaa... /bs/
2209. 14792 1792
Endel Tulving, Elements of Episodic Memory. Oxford University Press, 1983.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, 2022. URL https://arxxx
orq/ orq/abs/ 2210. 02399
Vikram Voleti, Alexia Jolicoeur-Martinnn. and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022. URL https::/arxiv.orr
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023. URL https://azxxi..og///aaaa. 06571
Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu. World- dreamer: Towards general world models for video generation via predicting masked tokens, 2024a. httttp:://rxxi..orrt 2401 09985
Y. Wang, D. Ma, and D. Cai. With greater text comes greater necessity: Inference-time training helps long text generation, 2024b. URL
11504
Jialong Wu, Shaofeng Yin, Ningya Feng. Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models, 2024. URL https://arxiv: axc/abs/2405.152
Jiannan Xiang. Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Pandora: Towards general world model with natural language actions and video states, 2024. URL https://
arxiv. abs /40/ 09455
Sherry Yang, Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schu- urmans, and Picter Abbeel. Learning interactive real-world simulators, 2024. URL https:
//arxiv.org/abs/23333 2ttp://b///3100011.1 06114
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy, Rerender a video: shot text-guided video-to-video translation, 2023. URL SRI.ttts:: http://wwwwvv.o..oa///// http///abb/23.... 07954
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang. Minheng Ni. Zhengyuan Yang. Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming, Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. Nuwa-xl: Diffusion over diffusion for extremely long video gener- ation, 2023. URL https: /tttps://axiivy org/ abs 220.//333..111
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn. and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021. URLhttps: htt////xxvvv.../// abs/ 995//99100 00.10.08890 10897
14

---

Published as a conference paper at ICLR 2025
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation, 2023. URL https://arxiv: www.aiii.rr............. abs/
10982
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models, 2023. URL htttps///axxat
abs abs/ 0tss/2211.11013
article ference,,,,e limes
amsmmath.mmsfonts.mm
Iphhhmss
boldOTIphvbxx
*arg max org max arg minarg min
sign Trgr
hyperref url algorithm algorithmic graphicx wrapfig xcolor enumitem colortbl
CONTENTS
Introduction
Related Works
3
SlowFast-VGan
3.1
Slow Learning
3.1.1
Masked Conditional Video Diffusion
3.1.2
Dataset Collection
3.2
Fast Learning
3.3
Slow-Fast Learning Loop with Temp-LoRA LoRA
3.4
Video Planning
3.5
Rethinking Slow-Fast Learning in the Context of Complementary Learning Systems
4
Experiment
Evaluation on Video Generation
4.2
Evaluation on Long Horizon Planning
5
Conclusion and Limitations
10
Contribution Statement
17
B
More Details about the Method
17
Dataset Statistics
18
More Experimental Details
18
Experiments Ablations and Variations of SLOOWFAS VGEN
19
15

---

Published as a conference paper at ICLR 2025
More Qualitative Examples
20
16

---

Published as a conference paper at ICLR 2025
A CONTRIBUTION STATEMENT
Yining Hong was responsible for all of the code development, paper writing, and experiments. She also collected the data for Minecraft.
Beide Liu contributed to most of the data collection with regard to Unreal data. He was in charge of setting up the Unreal Engine, purchasing assets online, writing the Python scripts for automate agent control, and recording first-person and third-person videos of Unreal data.
Maxine Wu collected the data of Google 3D Tiles. She was also responsible for the task setup of RLBench and the data collection of RLBench. She also curated part of the driving data.
Yuanhao Zhai wrote the codes for AnimateDiff, which was one of the baseline models.
The other people took on the advising roles, contributing extensively to the project by offering in- novative ideas, providing detailed technical recommendations, assisting with troubleshooting code issues, and conducting multiple rounds of thorough paper reviews. They provided valuable exper- tise on video diffusion models. Zhengyuan Yang, Yingnian Wu and Lijuan Wang were involved in brainstorming and critical review throughout the project. Specifically, Zhengyuan Yang pro- vided much technical support. Yingnian Wu came up with the idea of TEMP-LoRe for modelling episodic memory well the masked video diffusion model. Lijuan Wang provided valuable insights throughout the project.
B
MORE DETAILS ABOUT THE METHOD
B.1 PRELIMINARIES ON LATENT DIFFUSION MODELS
Stable Diffusion (Rombach et al. 2022), operates in the compressed latent space of autoencoder obtained by a pre-trained VAE Given an input zg, the process begins by encoding it into latent representation: 2 = Exxo) where E' is the VAE encoder function. Noise is then progressively added to the latent codes through a Gaussian diffusion process:
(<<(2(1)) W(2: /1 3, B,I)
fort 1...., where is the total number of diffusion steps and are noise schedule parameters. This iterative process can be expressed in simpler form:
C + vI EGE
where e .N 1). ā₂ = II ag and a₂ = 1 - 3r. Stable Diffusion employs an e-prediction approach, training neural network Eg to predict the noise added to the latent representation. The loss function is defined as:
E1 e [[]e oa 4, c)
Here, represents the conditioning (e.g., text), and denotes the neural network parameters, typi- cally implemented U-Net (Ronneberger al. 2013).
During inference, the model iteratively denoises random Gaussian noise, guided by the learned generate latent representations. These are then decoded to produce high-quality images consistent with the given textual descriptions.
Video diffusion models (Ho et al., 2022b) typically build upon LDMs by utilizing a 3D U-Net architecture, which enhances the standard 2D structure by adding temporal convolutions after each spatial convolution and temporal attention blocks following spatial attention blocks.
B.2 PRELIMINARIES ON LOW-RANK ADAPTATION (LORA)
2021 transforms the fine-tuning process for large-scale models by avoiding the need to adjust all parameters. Instead, it utilizes compact, low-rank matrices to modify only subset of the model's weights. This approach keeps the original model parameters fixed, addressing the problem of catastrophic forgetting, where learning can overwrite existing knowledge. LoRA
LoRA Hu et al.
17

---

Published as a conference paper at ICLR 2025
utilizes compact, low-rank matrices to modify only subset of the model's weights, therefore avoid- ing the need to adjust all parameters. In LoRA, the weight matrix W E Rmxx" is updated by adding learnable residual. The modified weight matrix W is:
W' = W + AW = W + AB
where A E Rmxe and B E R are low-rank matrices, and is the rank parameter that determines their size. In this paper, we denote the LoRA finetuning as the fast learning process and the pre- training slow learning process. The equation then becomes:
W W + AW Wsew + Winst + θ
(10)
where & corresponds to the pre-trained slow-learning weights, and é corresponds to the LoRA paraemters in the fast learning phase.
B.3 MODELSCOPET22 DETAILS
We base our slow learning model on Modelscope112V (Wang et al. 2023 Here, we introduce the details of this model.
Given a text prompt p. the model generates a video Upr through a latent video diffusion model that aligns with the semantic meaning of the prompt. The architecture composed a visual space where the training video Egt and generated video Upr reside, while the diffusion process and denoising UNet Eg operate in a latent space. Utilizing VQGAN, which facilitates data conversion between visual and latent spaces, the model encodes training video l/t [fi..... fr] into its latent representation Zgz [F([i,,-..
E(rr). During the training phase, the diffusion process introduces Gaussian noise to the latent variable, ultimately allowing the model to predict and denoise these latent representations during inference.
To ensure that ModelScopeT22 generates videos that adhere given text prompts, it incorporates text conditioning mechanism that effectively injects textual information into the generative process. Inspired by Stable Diffusion, the model augments the UNet structure with a cross-attention mech- anism that allows for conditioning of visual content based on textual input. The text embedding c derived from the prompt p is utilized the key and value in the multi-hear head attention layer, enabling the intermediate UNet features to integrate text features. The text encoder from the pre-trained CL.IP VIT-H/14 converts the prompt into a text embedding, ensuring strong alignment between language and vision embeddings.
The core the latent video diffusion model lies in the denoising UNet, which encompasses vari- ous blocks, including the initial block, downsampling block, spatio-temporal block, and upsampling block. Most of the model's parameters are concentrated in the denoising UNet cø. which is tasked with the diffusion process in the latent space. The model aims to minimize the discrepancy between the predicted noise and the ground-truth noise, thereby achieving effective video synthesis through architecture also includes spatio-temporal block, which captures complex spatial and temporal dependencies to enhance video synthesis quality. The spatio-temporal block is comprised of spatial convolutions, temporal convolutions, and attention mechanisms. By ef- fectively synthesizing videos through this structure, ModelScopeT2V learns comprehensive spatio- temporal representations, allowing to generate high-quality videos. The model implements com- bination of self attention and cross- attsenattntion mechanisms, facilitating both tross-modal interactions and spatial modeling to capture correlations across frames effectively.
denoising. ModelScope
C DATASET STATISTICS
We provide the dataset statistics in Figure 6
D MORE EXPERIMENTAL DETAILS
D.1 EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS
We utilize approximately 64 V100 GPUs for the pre-training of SLOWFAST-VGERT with a batch size of 128. The slow learning rate is set 5e-6, while the fast learning rate is 1e-4. Training
18

---

Published as a conference paper at ICLR 2025
Game: Minecraft
Unreal-FPV
Unreal-TPV
Unreal-Google 3D
Tiles
Unreal-Car
EPIC-KITCHEN
Open-X Embodiment
Other Robotics
Dalaset
HDD
Figure 6: Statistics of our Training Dataset.
videos of mixed lengths are used, all within the context window of 32 frames. During training, we freeze the VAE and CLIP Encoder, allowing only the UNet be trained. For inference and fast learning, we employ single V100 GPU. For TEMP-LORA, LoRA rank of 32 is used, and the Adam optimizer employed in both learning phases.
D.2 COMPUTATION COSTS
we show the computation costs with and without TEMP-LORA. While the inclusion of TEMA-LORA does introduce some additional computation during the inference process, the differ- ence is relatively minor and remains within acceptable limit.
In Table 3
Ours wo TEMP-LLORA
Ours w T'MM--LORA
Average Inference Time per Sample (seconds)
12.9305
13,8066
Inference Memory Usage (MB)
9579
9931
Table 3: Comparison of Computation Costs with and without TEMP-LORA
D.3 HUMAN EVALUATION DETAILS
In our human evaluation session for action-conditioned long video generation, 30 participants as- sessed the generated video samples (50 videos per person) based on three criteria:
Video Quality (0 to I): Participants evaluated the overall visual quality, considering aspects such as resolution, clarity, and aesthetic appeal.
Coherence (0 to 1): They examined the logical flow of actions and whether the events progressed seamlessly throughout the video, ensuring there were no abrupt changes or disconnections.
Adherence Actions (0 to I): Participants judged how accurately the generated videos reflected the specified action prompts, assessing whether the actions were effectively de- picted.
Each video was rated by at least three different individuals ensure reliability. The collected ratings were then compiled for analysis, with average scores calculated to assess performance across the different criteria.
E EXPERIMENTS ON ABLATIONS AND VARIATIONS OF SLOWFAST-VGER
We introduce several variations of SL.WFFAT--VGEE including:
19

---

Published as a conference paper at ICLR 2025
SCuts
SRC f
Our (w original TEMP-LORA)
0.55
92.24
Ours (wo Local Learning Rule)
0.36
90.27
Ours (wo Chunk Input)
1.24
90.01
Ours (wo/ Temp-LoRA)
1.88
89.04
Ours SLOWFAST-VGEN
0.37
93.71
Table 4: Scene Cuts and SRC Scores. Comparison of scene cuts and SRC scores for our method with and without Temp-LoRA.
Ours Chunk Input that only conditions single-frame frame images instead of previous chunk
Ours wo Local Learning Rule that samples over whole generated sequence for training TEMP-LORA, instead of using local inputs and outputs to train.
Ours w original TEMP-LORA that uses the original TEMP-LORA structure that were designed for long text generation.
We show the results below. From the table, full sequence also shows good performances. However, our observation indicates that this method tends over-smooth the generated sequences, leading blurry videos for later frames.
that SLOWWFSS-- VGEN trained over sampled
F MORE QUALITATIVE EXAMPLES
F.1
MORE QUALITATIVE EXAMPLES OF SLOW LEARNING
In Figure 7 and Figure g we include more qualitative examples with regard to slow learning.
F.2 MORE QUALITATIVE EXAMPLES OF FAST LEARNING
In Figure 9 and Figure 10 we include more qualitative examples with regard to fast learning.
20

---

Published as a conference paper at ICLR 2025
Switch to the Right Lane
Move to the Left
Put the Green Bowl in the Dish Rack
Right lane branch
Wash Plale
Pick Up the Apple
Back Off
Place the BLue Cube the Shelf
Rinse Knife
Initial
Ascend Higher Altiude
Descend Altiude
Adjust Position
Turn right
the MIlk Dairy
Back Off
Go Forward
Change Right Lane
Open Fridge
the nail with a hammer
Place the long bread in the sink
Pass the Intersection
Forward
Right
Place the apple fruit on the table
Open Cupboard
Back Off
Figure Qualitative Examples on Slow Learning. Part
21

---

Published as a conference paper at ICLR 2025
Pass the Intersection
Wash Hands
Pick up the orange fruit
Turn right
Run Straight Forward
Descend
Fly
Adjust Lateral the
Go Frwward
Place the apple fuuit in the BluIs bowl
Turn right
Open the Drawer
Wash Spoon
Tumn left
Move to the Right
Place the yellow cup in the oven
Tum on Tap
Shooting the Basketball
Pass the Crosswalk
Open the Drawer
Press the Button
Turn Left
Put the square bread the plate
Tum Right
Reverse the Car
Rinse Hands
Tum Left
Turn Right
Go Right
Figure 8: Qualitative Examples on Slow Learning. Part 2.
22

---

Published as a conference paper at ICLR 2025
== 0-32 right
t= 32-64 Run forward
== 64-96 forward
t= 96-128
1= 128-160
t= 160-192 left
Temp-LoRA
t= 192-224
t 224-256
== 258-288
I 268-320 right
t= 320-352 Forward
96-129 Go left
Go right
Run forward
Run forward
Back off
Go left
w/ Temp-LoRA
192-224 Run forward
224-256
256-288
288-320 Go right
320-352 Run Forward
Back off
Go left
= 0-32 Back Off
t= 32-64 Go forward
= 64-96 Move Right
= 96-126
128-160 Off
Move Left
wo/ Temp-LoRA
w/ Temp-LoRA
I= 0-32
32-84 Go forward
= 64-96 Back off
== 128 Go forward
128-160 Go fonward
t= 160-192 Back off
Go forward
wo TeMp-LoRA
w Temp-LoRA
Figure 9: Qualitative Examples on Fast Learning. Part 1. We mark consistent objects frames in green bounding boxes and inconsistent ones in red.
23

---

Published as a conference paper at ICLR 2025
1= D-32 Move right
32-64 Move right
64-96
96-128
128-160
160-192 right
192-224 Move right
224-256
Move
right
Temp-Leee LoRA
w/ Temp-ToRA
1 0-32
1 32-64
= 64-96 Back Off
= 96-128 Run Forward
1- 128-160 Run Forward
I 160-192 Back Off
t- 192-224 Back Off
Back Off
Back Off
Temp-LoRA
Temp-LoRA
t= 24-46
= 48-72 apple To the original location
== 72-96 Pick Dairy milk
t= 72-96 Place the Dairy milk
0-24 Pick up the Apple
Put apple
bowl
In the sink
Jemp-LoRA
w/ Temp-LoRA
Ga Forward
Farward
Back
Back Off
Go Forward
Back
Temp-Looe LoRA
w/ Temp-Lool LoRA
Figure 10: Qualitative Examples on Fast Learning. Part 2. We mark consistent objects frames in green bounding boxes and inconsistent ones in red.
24

---

Published as a conference paper at ICLR 2025
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, and et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512) 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023b. URL /atttpss/aaxx...orr//bb////////
08818
Xinyuan Chen, Yaohui Wang, Lingjun Zhang. Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction, 2023.
wttp:////.r.aabaa...2.0/ 20700
Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Man- dlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie. Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf. Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp. Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang. Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Ra- dosavovic, Isabel Leal, Jacky Liang. Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yun- liang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Nor- man Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart' Mart'in-Mart' in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola. Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suncel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vin- cent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang
25

---

Published as a conference paper at ICLR 2025
Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma. Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024. https:/:: http://://.aaai..oooaa//////////
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scal- ing egocentric vision: The epic-kitchens dataset. European Conference on Computer Vision (ECCV), 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100. Interna- tional Journal of Computer Vision (IJCV). 130:33-55, 2022. URL
org/1C.
10C7/s11263 021 01531-2
Yilun Du, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Joshua Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023.
URL httpt::ttt/attt http///wwiii...aaaaaa.../..
00111
Epic Games. Unreal engine. URL https:tpp//// www unrea www.aalennea. Lengine com
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models, 2023. URL
https: /ettps:/aarxive www./o/g/////////////o////..ee// 2302. 03011
songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer, 2022. /tttpss:://axxi www.exiiivoogg//o..
Yuwei Guo, Ceyuan Yang. Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models, 2023a. httltttps//aariv.ooo
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuming, 2023b.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos, 2022. URL
2205..225... 11495
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015,
03385
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- cls for high-fidelity long video generation, 2023. URL 13221
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text, 2024. URL pattps://arxiin org/ abs/
2403. 14773
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole. Mohammad Norouzi, Da J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022a. WwR.ihttps://exxiv.oom
2210. 02303
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022b. URL htRl/httts://arxx org abs 2204. 03458
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-l: 1: A generative world model for autonomous driving, 2023. URLhtttp htttts::///aaiiee www.or///aaa.... 17080
26

---

Published as a conference paper at ICLR 2025
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiv. abs /ww.//226.... 09685
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark and learning environment, 2019. URL https://arxiv. wwt.//org/aass.....
1909. 12271
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators, 2023. URL
13439
J. G. Klinzing, N. Niethard, and J. Born. Mechanisms of systems memory consolidation during sleep. Nature Neuroscience, 22:1598-1611 1619,, October 2019. doi: 10.1038/441999--0940006..3 Received 18 February 2019; Accepted 12 July 2019; Published 26 August 2019.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua Tenenbaum. Learning act from actionless videos through dense correspondences, 2023. URL htttts:/aaxxiv..o
abs 23300 08576
Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What learning systems do intelli- gent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512-334, July 2016. doi: 10. 10.1006/j.tics..o til. 20.6.00.66000 0..004.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation, 2023. URL
hbtp://b///2333 08320
James McClelland, Bruce L. McNaughton, and Randall C. O'Reilly. Why there are complemen- tary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419 457, July
1995. doi: 10.1037700333-255X 102.3.419.
Günther Palm. Neural associative memories and sparse coding. Neural Networks, 37:165-171, 2013.
PySceneDetect. Pyscenedetect. URU/tpps:/www.sseneeettt. com/ 03-03.
Accessed: 2024-
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxix: http:////axxvoooo///////////////
00020
Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene under- standing: A dataset for learning driver behavior and causal reasoning. In Conference Computer Vision and Puttern Recognition (CVPR), 2018.
Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation, 2024. URL https
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-106955 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, 2015. URL (RRLL'ttts/:/aiiiis http://wwww....o///////5//55.... 04597
Runway. Runway. URL httpt::t///etttt.ttt http::://ruuwwamm..o com/ Accessed: 2024-03-03.
Gladys C. Schwesinger. Review of "psychological development" by norman I. munn. Pedagogical Seminary and Journal of Genetic Psychology, 55xxxx-XxX, 1955.
27

---

Published as a conference paper at ICLR 2025
Uriel Singer, Adam Polyak, Thomas Hayes. Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: vixt-oo-video generation without text-video data, 2022. URL https: /ttp///aavvvvv/iaa///ieeeaaaaaaa org/ abs/
2209. 14792
Endel Tulving. Elements of Episodic Memory. Oxford University Press, 1983.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, 2022. URL httpsssst///ax
arg/abs/22110 02399
Vikram Voleti, Alexia Jolicoeur-Martnenn and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022. URL
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023. URL 06571
Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang. Xinze Chen, and Jiwen Lu. World- dreamer: Towards general world models for video generation via predicting masked tokens, 2024a. URLhttps: //tptpps///rxxv....// org/ abs /0lb/24500 2400 09985
Y. Wang, D. Ma, and D. Cai. With greater text comes greater necessity: Inference-time training helps long text generation, 2024b. URL htllhttss///xxi http://..wggaaaa/........
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models, 2024. URL https://arxiv.
org/abs/ 2405. 15223
Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Pandora: Towards general world model with natural language actions and video states, 2024. URL https://
wrxiv.org/aall /2004 09455
Sherry Yang, Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schu- ummans, and Pieter Abbeel. Learning interactive real-world simulators, 2024. URL https:
wwt. abggg//bs//00 22310 ..1.1.1.0...... 06114
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation, 2023. URL/LLpp://axxvvvoo///// 20..bb///33...00 07954
Shengming Yin, Chenfei Wu, Huan Yang. Jianfeng Wang, Xiaodong Wang. Minheng Ni, Zhengyuan Yang. Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming. Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. Nuwa-xl: Diffusion over diffusion for extremely long video gener- ation, 2023. URL http:::tt//aattt
erq/ abs /2002.22333..2 12346
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021. URLhttts:://axxiv org/
abs//9100 00..1.118999 10897
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation, 2023. URL htttts://axxi..co. abs/
231. 10982
Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv. Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models, 2023. URL htt.https: prttps://ase http:::s//rxxiv.roo abs/ 22 1018
28


Reviews:
[{"review_id": "98h4wknwrf", "reviewer": "Reviewer_iFRv", "summary": "The paper introduces SLOWFAST-VGEN, a dual-speed learning framework for action-driven, long video generation inspired by the complementary learning system in human cognition. This model integrates slow learning of world dynamics with fast learning for episodic memory storage, aiming to address challenges in generating consistent, coherent long videos. The slow learning component utilizes a masked conditional video diffusion model, while the fast learning component leverages LoRA to store episodic memory. The authors evaluate the model on various datasets and metrics, achieving superior FVD scores and scene consistency compared to baseline models.", "questions": "1. How are the parameters optimized across the slow and fast learning phases to ensure smooth interaction without redundancy? Could alternative approaches, such as dynamically adjusting memory encoding based on the video’s complexity, further streamline the integration?\n2. For tasks that require nuanced, multi-step interactions (e.g., tasks involving conditional actions or dependencies between actions), how does SLOWFAST-VGEN manage action dependencies across time steps? Are there specific mechanisms to ensure that complex sequences remain consistent and contextually appropriate?\n3. What do you think about the hierarchical learning paradigm? Such as the slow/fast learning v.s. long-horizon key-frame generation + show-horizon dense-frame generation. This is just a question, not need to conduct experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novelty in Dual-Speed Learning**: By combining slow learning of world models and fast episodic learning, SLOWFAST-VGEN represents an innovative approach, mimicking cognitive memory systems effectively.\n2. **Comprehensive Benchmarking**: Evaluations include diverse tasks and metrics, reinforcing the model’s robustness across applications.", "weaknesses": "1. **Limited Generalization**: SLOWFAST-VGEN’s performance may drop in scenarios outside its training domains, such as novel environments or complex, unforeseen actions, limiting its adaptability in dynamic real-world applications.\n2. **Memory Inconsistency Over Long Sequences**: TEMP-LORA effectively stores episodic memory within single episodes but struggles with consistency across multi-episode or long-term tasks, potentially leading to fragmented recall in extended sequences. Since the authors mention that the slowing learning for world modeling can capture general dynamics, some long-horizon tasks need to be evaluated, such as moving to the right and then to the left.\n3. **High Computational Cost**: The reliance on a diffusion model for video generation demands significant computational resources. Although using LoRA, the two-stage training. What about latent diffusion models for the slow learning, while video diffusion models for the fast learning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SSXbBJKZD1", "reviewer": "Reviewer_C2ZJ", "summary": "In this paper, the authors focus on the problem of long term video generation. To this end, they propose SlowFast-VGen, a model which leverages episodic memory (using a LoRA module from long text generation) to improve context utilization over longer time horizons. The authors also argue that the approach has connections to Cognitive Science. They then evaluate their approach on multiple datasets including RLBench and Minecraft. They demonstrate convincing quantitative results that beat baselines on metrics measuring long term consistency.", "questions": "1. The approach is complex. Are there any possible ways to streamline the model while still leveraging the Slow/Fast concept?\n\n2. How would this approach deal with a complex \"real world\" dataset of long videos beyond just constrained domains such as Kitchens or Car Driving?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses a serious failing in contemporary approaches to video modeling. Current video models cannot scale well to long time horizons, and this paper shows promise in advancing this capability.\n\n2. The paper is well written and easy to understand.\n\n3. Experimental evaluation is thorough, and the datasets used is reasonably diverse. The quantitative results are convincing.", "weaknesses": "1. The proposed method is quite complex, requiring multiple working parts (Slow vs Fast modules).\n\n2. I think the paper might benefit from more complex datasets. Datasets from video games, robotic simulation, and car driving are used, but these are not particularly diverse. Training/testing on a dataset such as Walking Tours (Venkataramanan et al 2024) would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LWBSnyvJ9", "reviewer": "Reviewer_iXgc", "summary": "In this paper, the authors tackle the challenging action-driven video generation task. They approach it by dividing the model and training into a slow and fast learning loop, reflecting the changes in the  dynamics in the world model and in individual episodes. This is achieved by adding a temporal LoRa to the base architecture, which is also used during inference. While minimally increasing the inference time, this temporal-LoRa leads to an improved generative performance and temporal consistency.\nTo train the model the authors create a new large scale video-action-annotation dataset. The model performance is evaluated on video generation tasks and long-horizon planning.", "questions": "1. Is there a limit of the temp-lora in terms of the capacity of trajectories of individual objects in the scene if their motion is not the same as the global motion?\n2. I did not find the number of diffusion steps T used in the experiments, please provide these numbers and any other hyperparameters used if different than the base model.\n3. How is the performance of the model on dynamic objects in a scene if you re-visit an area, e.g. moving objects like cars or persons in an early episode that you re-visit later?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well structured, and additional details in the appendix help to clarify some aspects.\n2. The authors introduce a bio-inspired interpretation of a slow-and fast learning loop to improve long video generation by maintining an episodic memory, and link the inspiration to the hippocampus-cortex learning hypothesis. I like the independence of the lora loop from the action label embedding and treating the previous and next chunk as a single continuum, making it easier to assess the actual impact of this architecture change.\n3. A new dataset for benchmarking action-driven video generation models is introduced, leveraging existing datasets and adding new (mainly video game engine) data and labels. The authors compare their model with several s.o.t.a. models and introduce a new metric for scene consistency.\n4. The model is benchmarked using several metrics, including one that requires multi-sampling with human labeling.", "weaknesses": "1. The interpretation of  fast and slow memory or training loops is a staple e.g. in generative replay in the continual learning domain (including often citing the hippocampus inspiration), as well as several RNN-like architectures with slow and fast memory. A link to these previous works (even if just in the appendix) could further improve the paper.\n2. Novelty: the authors cite some previous work on the LLM domain as the main source of inspiration for this usage of temp-lora. The term temp-lora and its application in the video generation domain was used in previous works such as [1] (which is only a preprint, but given it is from feb 2024 still relevant for this review), a statement how their lora application differs would be appreciated. Given these works are only preprints it does not mean they take away from the novelty of this submission. It is just to get a better understanding of the differences in how lora is applied. Even if it is the same technique it would be interesting to know if the authors think the chosen task and datasets are better suited for the technique compared to the taks & datasets in the previous works.\n3. I would like to see more architectural and (hyper) parameter details of the model used in the experiments (esp. The Unet + Lora), even if it is based on existing pretrained models, and how the hyperparmeters were chosen (if different from the base models).\n\nOther inputs (not weaknesses) that may be used to improve the paper: \nThere seems to be repeating text in appendix A.2 Preliminaries On Low-Rank Adaptation\n\n[1] Ren, Yixuan, et al. \"Customize-a-video: One-shot motion customization of text-to-video diffusion models.\" arXiv preprint arXiv:2402.14780 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uigQP9zmJa", "reviewer": "Reviewer_bXBH", "summary": "This paper proposes a slow-fast structure for retaining episodic memory in long video generation. The “slow” video model follows similarly to prior works in masked conditional video diffusion models, where x prior frames are conditioned on to generate y future frames. To incorporate longer-context memory when generating multiple chunks in a row, the authors introduce temporal LoRA parameters that are updated during inference time to store episodic memory. For more general long-context generation tasks, the paper additionally proposes a slow-fast training algorithm.", "questions": "* It would be useful if the authors could include some way to view the original video samples (not just snapshots of frames in the paper). Perhaps with an anonymous site?\n* How long are the generated videos in seconds? Most figures show timestamps, but framerate is not mentioned in the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is generally easy to follow, and well written\n* The proposed algorithm is interesting, and could be a promising direction for incorporating memory through test-time training in neural networks\n* Experimental results on long videos show better consistency than baseline methods, and without using their temporal LoRA module.\n* The authors introduce a collection of action-conditioned video that could be useful for future related research", "weaknesses": "* The purpose of the “slow” video baseline comparisons are a little confusing to me. Is this not just showing that training on specifically curated action data results in a model that can better generate conditioned on actions?\n* The authors have shown some benefits in retaining long-term memory over the course of multi-chunk video generation. However, I am still a little doubtful at the long-context abilities of this algorithm if used at scale. It seems difficult to propagate / learn abstract representations in long term memory that may be useful for long-term video generation, especially since the temporal LoRA is updated using a pixel-wise diffusion loss and thus biased to memorizing a specific instance of a scene. How would this model perform in more difficult long context tasks (e.g. remembering something seen 10-30s ago, or trying to localize position / orientation in 3D space), such as the datasets introduced in [1]? Or, in a more simpler environment, making one full rotation, and then randomly turning left and right (the first rotation should newly generate the scene, and afterwards only requires accurate scene retrieval from memory). \n\n\n[1] Yan, Wilson, et al. \"Temporally consistent transformers for video generation.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)