WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception
Authors: Zhiheng Liu
Xueqing Deng
Shoufa Chen
Angtian Wang
Qiushan Guo
Mingfei Han
Zeyue Xue
Mengzhao Chen
Ping Luo
Linjie Yang
Venue: NeurIPS 2025 poster ()
Keywords: long video generation
world models
Primary area: deep_learning

Abstract: Generative video modeling has made significant strides, yet ensuring structural and temporal consistency over long sequences remains a challenge. Current methods predominantly rely on RGB signals, leading to accumulated errors in object structure and motion over extended durations. To address these issues, we introduce WorldWeaver, a robust framework for long video generation that jointly models RGB frames and perceptual conditions within a unified long-horizon modeling scheme. Our training framework offers three key advantages. First, by jointly predicting perceptual conditions and color information from a unified representation, it significantly enhances temporal consistency and motion dynamics. Second, by leveraging depth cues, which we observe to be more resistant to drift than RGB, we construct a memory bank that preserves clearer contextual information, improving quality in long-horizon video generation. Third, we employ segmented noise scheduling for training prediction groups, which further mitigates drift and reduces computational cost. Extensive experiments on both diffusion and rectified flow-based models demonstrate the effectiveness of WorldWeaver in reducing temporal drift and improving the fidelity of generated videos.

Weaver Generating Long-Horizon Video Worlds via Rich Perception
Zhiheng Liu Xueqing Deng", Shoufa Chen', , Angtian Wang', Qiushan Guo?, Mingfei Han?, Zeyue Xue Mengzhao Chen', Ping Luo't, Linjie Yang"t
The University of Hong Kong. "ByteDance Seed
Abstract
Generative video modeling has made significant strides, yet ensuring structural and temporal consistency over long sequences remains a challenge. Current methods predominantly rely on RGB signals, leading to accumulated errors in object structure and motion over extended durations. To address these issues, we introduce WorldWeaver, robust framework for long video generation that jointly models RGB frames and perceptual conditions within a unified long-horizon modeling scheme. Our training framework offers three key advantages. First, by jointly predicting perceptual conditions and color information from a unified representation, it significantly enhances temporal consistency and motion dynamics. Second, by leveraging depth cues, which we observe to be more resistant to drift than RGB, we construct memory bank that preserves clearer contextual information, improving quality in morghoorioon vidco generation. Third, we employ segmented noise scheduling for training prediction groups, which further mitigates drift and reduces computational cost. Extensive experiments on both diffusion- and rectified flow-based models demonstrate the effectiveness of WorveWeaver reducing temporal drift and improving the fidelity of generated videos. Page could be found here.
1 Introduction
Long-horizon video prediction [10, 76, 11, 77, 17, 45, 46] is fundamental challenge for world modeling, as it requires models to internalize and reproduce the causal laws that govern scene dynamics over time. Although generative video modeling [65, 34, 34, 48, 47, 49], has made rapid progress and shows impressive performance on short sequences less than 10 seconds, preserving realism and coherence over longer durations remains a significant challenge. Current approaches often struggle to depict correct motion and shape in long-horizon predictions, resulting in severe degradation including objects warp, motions drift, and violations of physical constraints as time progresses. These challenges highlight the difficulty of preserving both dynamics and temporal consistency in long video generation, which critical for structured and predictive understanding of the world.
Most generative video models nowadays rely exclusively RGB information training signals, optimizing pixel-level reconstruction objectives. This causes them to favor color and texture over structural and dynamical attributes, such motion trajectories, object geometry, and scale relationships [9, 40, 18]. In other words, training solely on RGB information can lead to the aforementioned instabilities. Such instabilities can be magnified over long sequences small frame-wist errors accumulate and compound, resulting in pronounced temporal drift and structural degradation.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
(b)
Prompt: woman walks down the street, smiling as she puts on her sunglasses. She stops to wave hello, then turns around and walks away.
Figure 1: WorldWeaver vs. existing approach on Long-Horizon Video Generation. Compared to other methods (b), WorldWeaver (a) achieves superior temporal consistency and motion quality in long-horizon video generation.
To address these issues, as shown Fig. 1, we introduce WerveWeaver, a robust framework for accurate Long-horizon video prediction that seamlessly integrates joint modeling of RGB frames and perceptual signals. Our approach employs a perceptual-condiiione next prediction scheme to achieve precise and stable long-video generation. Our method encourages the model capture just color, but also structural and motion-related cues, thereby improving the robustness of the generated outputs. To achieve this, we incorporate multiple perceptual signals-such as video depth and optical flow-into a joint training framework. Extensive experiments confirm the effectiveness of using these auxiliary signals. Inspired by the observation that depth is more stable than RGB under temporal drift, introduce a memory bank that stores reliable historical information from previous frames to enhance temporal consistency. This allows our model to maintain clearer long-term context and effectively mitigate the accumulation of drift in RGB predictions. In contrast, prior methods [10, 60, 77, 24] often inject substantial noise into historical information prevent drift, which diminishes the utility of long-range context. Additionally, noise levels to separate segments during training and constructing delayed denoising prediction groups during inferenceto further mitigate drift and reduce computational cost.
adopt segmented noise scheduling strategyassigning different
To evaluate our framework, we verify its effectiveness on both diffusion-baan based (CogVideoXX 1.5B) and rectified flow-based WWan2.1-1.3BB models on a robotic-arm manipulation dataset and a general- purpose video dataset. Extensive experiments demonstrate that our method substantially improves the stability and consistency of long-ho horizon video generation, effectively reducing temporal drift and structural distortions across diverse scenarios.
Our contributions can be summarized as follow:
Systematically exploring the role of image- perceptual condition, such as depth and optical flow, in enhancing long-horizon video generation auxiliary signals.
Proposing a unified framework that integrates perceptual conditioning and memory mechanisms for robust long-horizon video prediction.
Extensive validation across different generative models and datasets, including both general general-purpose and robotic manipulation domains, highlighting the potential of our approach as foundation for scalable world models.

---

2
Related Works
Long video generation. With recent advances in video diffusion models [5, 34, 63, 38, 12], long video synthesis has garnered increasing attention for applications such as storytelling and simulations. Existing approaches span GAN-based methods [22, 41], training-free techniques [52, 64), distributed inference [61], autoregressive frameworks [74], and hierarchical strategies [31, 73]. Among these, FreeNoise [53] extends diffusion models by rescheduling noise without fine- tuning: Streaming T2V TVV [32] integrates autoregressive modeling with memory modules for improved coherence; and TTT [13] adapts models at test time using specialized layers. LCT [24] enhances pre-trained models with 3D RoPE and asynchronous noise scheduling to produce scene- peene-consistent multi-shot videos, while FAR [23] employs frame-level autoregressive modcling with FlexRoPE and context strategies for efficient long video generation. However, most existing methods rely solely on RGB information from historical frames, leading to cumulative structural and motion errors. In contrast, our approach incorporates perceptual signalssuch as depth and optical flowfrom historical frames to more effectively guide future frame generation.
Generative model for world simulation. World models, originating from foundational work [26], have evolved from RNN-based latent dynamics [27-29] generative approaches such as diffusion- based [2, 15, 62, 19, 1] and autoregressive models [6, 44, 56, 39, l]. Unlike agent-centric prediction, physical-world simulation emphasizes ohysics-informed realism for real-world applications. Early efforts aim to enhance simulator outputs: RL-CycleGAN [54] refines rendered images while aligning utility through Q-values, and RetinaGAN [33] preserves object features in simulated scenes for reinforcement learning. Recent diffusion [65, 57, 4] and autoregressive diffusion [10, 60, 74] frameworks surpass GANs by producing figher-fidelity more temporally consistent, and realistic dynamic outputs. These advances enable video generation models to serve as learnable simulators, powering phyiicdggr simulation applications such as game environments [6, 62, 39, 7, 75, 14, 71J, autonomous driving [35, 37, 51, 69, 20, 21J. and robotic control [3, 16, 8, 30, 25, 78]. However, current video generation frameworks often prioritize color and texture over robust modeling of dynamiis and spatial relationships [40, 9]. During inference, they frequently struggle to incorporate real-world knowledge beyond historical RGB data, limiting their ability to simulate complex interactions accurately.
3 Methods
In this section, we first introduce the preliminary and background on video diffusion models (Sec. 3.1). To enhance structure and motion modeling capabilities in video generation, we propose to jointly learn perceptual cues, such as depth and optical flow from video data that captures rich spatial structures and temporal dynamics (Sec. 3.2). We further propose perception conditione long-horizon generation framework, which leverages a perceptual memory bank and group-wise noise scheduling to maintain robust context and mitigate drift in long video prediction (Sec. 3.3).
3.1 Preliminary
Video diffusion models offer powerful framework for generating high-quality videos from conditional inputs. These models generate coherent vidco sequences by learning to reverse a noise-injection process, progressively refining random noise into structured outputs.
We adopt flow matching [43] which offers a deterministic alternative to stochastic diffusion processes. Flow matching constructs continuous trajectory from a noise distribution to the target video distribution, conditioned on an input y, representing the conditioning signal (e.g., a text prompt). Formally, given a clean video sample condition y, the interpolation at timestep E [o, 1] is: It = tx + (1 f) The model learns the conditional velocity of this trajectory, defined as: U, = dei = 21 Zo. The training objective minimizes the discrepancy between the predicted velocity u(2,, t; t) and the true velocity:
XT*CCXX
noise sample xo No0, I), and the
de
(())=
[lu(2,, y, t; v.
(1)
where denotes the model parameters and y the conditioning input. During inference, an ordinary differential equation (ODE) solver integrates the velocity field to generate videos from

---

Streaning decoding
Diffusion Transformer
Memory bank
Prediction groupt
Denedee
3b VAE
Timdiis
Somple froe 4 grops
Figure 2: Framework of Weaver Given an input video, RGB, depth, and optical flow signals are encoded into a joint latent representation via 3D VAE. The latents are split into memory bank and prediction groups for the Diffusion Transformer. The memory bank stores historical frames and is excluded from loss computation; short-term memory retains few fully denoised frames for fine details, while long-term memory keeps depth cues noise-free and adds low-level noise RGB information. During training, prediction groups are assigned different noise levels according to the noise scheduler curve, aligned with the noise scheduling used during inference.
noise, guided by y. To mitigate the computational complexity of high-dimensional video data, models employ pretrained variational autoencoder (VAE) to compress video sequences into a latent space. Given video sequence X E F=xxxxx* where F, C, H, and W represent the number of frames, channels, height, and width, respectively, the VAE maps it to a latent representation 2g xexxxx This compression reduces the temporal and spatial dimensions while preserving essential features, facilitating efficient optimization in the latent space. Diffusion Transformer serves as the primary network architecture, leveraging attention mechanisms to model long-range spatio-temporal dependencies.
3.2 Enhancing Video Generation with Perceptual Conditions
Recent studies [40, 9] reveal critical limitation in current diffusion-based video generation approaches: the learning is prioritized towards color information, leading to a neglect of essential attributes such as motion, relative size, shape, and spatial relationships. This bias hampers the ability of models to capture the dynamic and structural complexities video data, particularly next-frame prediction frameworks for long video generation. Here, distortions in shape, size, or motion in individual frames can trigger rapid error accumulation, lcading to incoherent sequences. To address this, introduce unified modeling framework that integrates RGB data with perceptual conditions, specifically video depth and optical flow. Additionally, we evaluate the impact of various perceptual conditions, including video segmentation, with findings detailed in Sec. 4.3.
We first describe the process of preparing depth and optical flow data for our training videos. For depth estimation, we employ Video Depth Anything [68] to generate initial depth maps for each frame the input video. Then we use DepthAnything [70] to compute relative depth across frames through least-squares optimization. The resulting depth sequences are normalized to form match the dimensionality of the VAE input, we repeat the single channel three
D E xxx times, giving D E R f'3xx Hf xW This depth representation is then compressed into latent space using a variational autoencoder (VAE): 2d = ED), where denotes the VAE encoder.
Next, we employ SEA-RAFT [66] to extract optical flow between consecutive frames of the input video, producing a displacement field O E x where O(u, v) represents the displacement of the pixels between frames. To encode this into format compatible with our framework, we convert O into an RGB representation by computing the magnitude and direction of
motion of each pixel, defined as m min (1, oVH*4W=
v//4422 and a arclan 2(0, u), with a 0.15,

---

First Frame
60 Frame
120th Frume
180 Frame
RGB only
Depth only
Joint
Figure 3: Visualization of drift resistance. We construct video data by repeating single image during training and include "static" in the prompt. When the model set to output only RGB, severe structural degradation and color distortion occur the number of frames increases. In contrast, when outputting only depth, the structure remains well preserved. Jointly outputting both RGB and depth significantly alleviates visual drift. See experimental details in supplementary materials.
following VideoJAM [9]. Here, m determines pixel opacity, and assigns color based on the motion direction. To align with the VAE input, we pad O with a zero frame to match the temporal dimension of the input video, forming O IR Xx3 This flow representation is then compressed into latent space using the VAE encoder: 2o {O)). Note that we use "synthetic" depth and optical flow data from open source predictive models, since obtaining ground-trute -tuuh depth/flow is not possible with large-scale web videos.
To integrate appearance, depth, and optical flow into unified latent representation, we extend the input and output layers of the pretrained model to accommodate the additional channels for depth and flow. For the RGB input channels, we initialize the model branch from the corresponding pretrained weights to reduce training cost, while the channels corresponding to depth and flow are randomly initialized to allow the model to adapt to these modalities during fine-tuning. As shown on the left of Fig. 2, after encoding, the latent representations of the video, depth, and optical flow are concatenated along the channel dimension training, we add noise to this joint latent at timestep ( t E(O, 1] to obtain z,. We extend the training objective to jointly predict all three modalities, defined as:
form a joint representation z = [Z., 2d, 2.]. During
((0)= Ez, -o..
|u* (z,, 2
where ut = [ăg, Eds 2,] denotes the predicted latents for RGB, depth, and optical flow, respectively, and uz [u," uy of' represents the target velocities. The loss computed across the channels corresponding to the modalities. Unlike the complex sampling-stard adopt a simpler approach by simultaneously leveraging RGB and current perceptual conditions to guide future frame generation, as detailed in the next section.
guidance in VideoJAM [9], we
3.3 Long-Horizon Generation Framework
Building on the unified modeling, we now describe how effectively leverage historical frame contexts, including appearance and perceptual conditions, to achieve stable long video generation. Our design is informed by following key insights: (1) During training with teacher forcing [67], the model is conditioned on ground-trutt -tuth frames inputs. However, during inference, it must rely solely on its own previously generated frames. This discrepancy between training and inference can lead to the accumulation of errors over time. To mitigate this issue, we adopt non-causal sampling head combined with delayed denoising [60], which allows the model to refine its predictions using additional context beyond the immediate past. (2) As shown in Fig. 3, depth prediction exhibits
5

---

greater temporal stability compared to RGB generation. When the model is trained to output only RGB, we observe rapid degradation in both color and shape over time. contrast, depth outputs alone show significantly less drift. A key reason for this robustness is that depth is invariant to illumination, shadows, and surface textures, which introduce high variability in RGB signals and are difficult to model consistently over long sequences. This invariance reduces the visual complexity the model must reproduce, thereby slowing down error accumulation. Furthermore, jointly predicting RGB and depth substantially improves RGB quality, suggesting that the geometric constraints provided noise
by depth help stabilize the appearance generation; (3) In contrast to the random scheduling used in Diffusion Forcing [10], we adopt a pre-defined, group-wiss noise schedule that more closely aligns training with inference behavior. This structured scheduling strategy reduces training instability and improves overall efficiency by providing more consistent temporal dynamics across samples. Detailed results and comparisons are presented in Sec. 4.2.
Specifically, we partition the inputs to the Diffusion Transformer into two parts: memory bank and prediction groups. The memory bank consists of both short-term and long-term memory. For short- term memory, we select small set of historical frames that are adjacent to the current frame. Since figh-frreuency details are typically reconstructed in the later denoising steps, we set the noise level of these frames to zero, allowing model to preserve fine-grained texture information. For long-term memory, these historical frames contribute less to the current frame's 's fine details. Although depth perception (as shown in Fig. 3) helps stabilize shape and size, RGB color drift can still occur. To bridge the training- infernncce gap, we apply a low-level noise tm to the RGB and optical flow channels of long-term memory, while keeping depth frames un-noised to provide stable global structural guidance. Without perceptual conditions, a high noise level (e-g.. m. = 0.3) is typically required for these channels, but this provides limited useful information. During training. f. is randomly sampled from 0.7 to 0.9 to enhance robustness, and during inference, it is set to 0.8. Note that a higher t. corresponds to a lower noise level, consistent with our preliminary.
For the prediction groups, we assign noise levels in a structured manner as follows: We divide the input frames into G' consecutive groups, where G is the total number of groups. For each training sample, we sample an index uniformly from the interval [1000- G
1000
1000). For each group
(k 0, 1, 1,.1........ G 1), the noise index is defined as:
1000
i 1~ 1000 1000
index - k G
1000
The corresponding sigma and timestep for each group obtained from the noise scheduler using index,, the training timesteps is set to 1000. This scheduling ensures that the noise intervals are evenly distributed among the G groups, and each group receives noise levels sampled from different segments of the scheduler curve. During inference, the groups are denoised sequentially in a streaming manner, with noise levels decreasing as each group is processed.
4 Experiments
4.1 Experimental Setup
Datasets. To comprehensively evaluate the effectiveness and generalizability of our proposed method, we conduct experiments using two publicly available foundation models: 11.2.1 [63] (flow-based), trained on an internal ceneral-purposs video dataset at a resolution of 480 832; and Cog VideoX222 [72] (diffusion-based)) trained on the in-the-wild DROID robotic manipulation dataset [42] at resolution of 480 x 720. The general-purpose dataset contains 300K raw videos, segmented into 1 million short clips, while the DROID dataset consists of 200K successful robotic operation clips. This setup allows us to assess the feasibility of our approach on both flow-based and diffusion-basse models, as well as its potential as long-horizon world model in both general and specialized domains.
Training. All training is conducted on 32 NVIDIA A100 GPUs with a learning rate of le-4, using the AdamW optimizer [50], with GPU batch size of 4. The W.n2.1-1.3B model is trained for 50K iterations while the C2 model is trained for 15K steps. To preserve text-to-video performance, we apply the same noise level to every frame in 10% of the training steps.
Evaluation. To assess video generation performance, we utilize VBench [36], framework that evaluates models across disentangled dimensions, grouping metrics into two categories: consistency

---

man takes a sip of coffee, then receives phone call and chats happily.
Time
He stands up from his seat, begins to walk,
and then starts dancing.
A robot arm picks up blue cup from the sink area and places it tray,
Time
then picks up orange cup places it tray.
then picks up the green one places it tray.
Figure 4: Generated Long- Morg-oorizon Video Results of WorldWeaver. WorldWeaver demonstrates strong generalization, as it can be applied to various base models. These results highlight not only the effectiveness our method, but also its potential as a versatile and extensible world model across diverse domains.
and motion. Consistency metrics assess per-frame subject and background coherence, while motion metrics evaluate the amount and temporal coherence of generated motion. We also evaluate a start-end quality contrast metric to measure potential drifting. Specifically, we compute the absolute difference in image quality metric between the first 5 seconds and the last 5 seconds of each video. A larger value of A quality indicates greater discrepancy between the beginning and end of the video. drift
4.2 Comparisons
Visualization of WorldWeaver. We present 30-second and 20-second long video generation results of our model on both general-purpose and robotic manipulation datasets in Fig. Due to space limitations, additional visualizations and comparisons with existing methods are provided in the supplementary materials.
Quantitative experiments. We compare our model with existing open-source methods capable of generating long videos, including S2V [32], Caus Vid (Wan) [74], SkyReels-V2 1.3B [11], and Magi 4.5B [59]. Specifically, we use ChatGPT-40 to generate initial images and corresponding text prompts, each containing 4-6 actions, which are used to guide video generation. These prompts correspond to videas of 20-30 seconds in length. As shown in Tab. 1, WorldWeaver surpasses the Magi 4.5B on drift and background consistency metrics. These results are partly influenced by differences in training data and engineering optimizations. In the next section, we provide a more detailed comparison under the same settings. addition, we conduct user study to complement the automatic evaluation metries, with further details available in the supplementary materials.

---

Table 1: Comparison of with other open-source models. Note that the motion degree metric is omitted, as nearly all 20-30 second videos are judged as true for this criterion, making it uninformative.
Methods
Subject consistency
Background consistency
Image quality
Motion smoothness
Quality
Streaming T2V
79.34
84.32
0.47
0.73
0.23
CausVid
84.71
88.43
0.54
0.82
0.16
SkyReels- V2
87.05
90.07
0.61
0.86
0.11
Magi
88.71
90.32
0.62
0.89
0.09
Ours Discussion-of long context design. Due differences in the dataset, madetttmmeelaeae
87.34
90.49
0.59
0.87
0.07
training
resources, and the use of post-processing techniques for long videos among existing open-source models, direct comparisons are not entirely fair. Therefore, We evaluate several relevant strategies for long-video generation using the same base model and dataset: Repeating image-lo-vided (I2V), which applies zero noise to the first frame and identical random noise to all subsequent frames during training, then concatenates the model's final output back into the input sequence extend video length; Caus Vid [74], which adapts the standard full-attention mechanism to a causal-only attention variant using the official implementation; History Diffiusion [60], which feeds fully noised historical frames into the unconditional branch of classifier-free guidance to reinforce long-term memory; Rolling Diffusion [58], which applies linearly decreasing, frame-wist wise noise schedule throughout the sequence: Diffusion Forcing [10], which applies random noise to each frame during training and delays the denoising of past frames during inference; and Ours (RGB-ont) and Ours BGGB+Perceptuall which use the memory bank and the prediction groups to model either RGB information alone RGB and perceptual channels jointly. Since FramePack [76] requires pre- generating the first and last frames, is not well suited for scenarios with substantial changes, such robotic manipulation dynamic scenes, and is thus not considered in evaluation. It is worth noting that we also discuss the convergence time required for each algorithm, defined as the training duration after which further improvements in model performance are longer observed.
Table 2: Comparison with relevant long-horizon methods. We compare with other relevant methods across the global metrics, drifting metric. The tests are conducted with DogVideXX-2B on robotic manipulation dataset, with experiments performed using 16 A 100 GPUs.
Methods
Subject consistency
Background consistency
Image quality
Motion smoothness
A
Quality drin
Training steps
12V
90.23
91.36
0.51
0.64
0.24
CausVid
90.43
91.56
0.53
0.67
0.19
History Diffusion
89.78
91.92
0.57
0.68
0.09
70K
Rolling Diffusion
88.74
91.09
0.58
0.70
0.05
20K
Diffusion Forcing
88,67
91.23
0.61
0.72
0.06
80K
Ours w/o perceps
89.57
91.51
0.55
0.67
0.15
30K
Ours
90.92
92.39
0.60
0.75
0.07
30K
As depicted in Tab. 2, our key findings are as follows: (1) Our approach achieves the best performance on most metrics, particularly terms of consistency, while performing comparably the best methods on other metrics. (2) Training with independent random noise schedules for each frame, which adopted by Diffusion Forcing and History Diffusion, requires more training steps for convergence. In contrast, WorldWeaver reduced the training time with the improved training strategies with aligned training and inference noise schedules. (3) While delayed denoising of historical frames mitigates drift, the lack of clean historical frames guidance compromises video consistency. (4) Compared to MGB-only modeling. unified modeling with perceptual conditions enhances ver-frame prediction accuracy, allowing the historical frames of the memory bank to maintain low noise level, thereby providing effective guidance for subsequent predictions and improving overall consistency.
4.3 Ablation studies
Impact of perceptual conditions. We assess the contributions of various perceptual conditions to improving different aspects of video generation. In addition to the video depth and optical flow

---

adopted in our final framework, we also explore joint modeling with video segmentation, which is derived using SAM2 [55] by retaining up 10 masks with the highest confidence scores for each video. To eliminate the influence of drift, we conduct these experiments on short 5-second videos.
Table 3:
Effects of various perceptions. We evaluate the impact of combining RGB with individual perceptual conditions, as well as jointly using multiple perceptual conditions together.
Subject consistency
Background consistency
Dynamic degree
Motion smoothness
Perceptions
92.72
94.76
0.62
0.74
w/ depth
94.87
95.14
0.70
0.82
w/ seg
94.01
94.83
0.65
0.70
w/ flow
92.48
94.22
0.74
0.81
w/ depth & seg
94.93
95.17
0.68
0.77
w/ depth & flow
94.76
95.23
0.73
0.85
As shown in Tab. 3, we find that video depth improves both consistency and motion, while optical flow mainly enhances motion. Segmentation contributes more to consistency but slightly reduces motion quality. Since both depth and segmentation capture similar shape and size information, combining them does not further improve consistency. In contrast, jointly modeling depth and optical flow leads to better motion performance. Compared to VideoJAM [9], which relies primarily on optical flow with complex sampling guidance, our approach demonstrates that conditioning on video depth yields greater improvements in temporal coherence and structural fidelity, with optical flow providing complementary benefits.
Noise level of memory bank. We further analyze the impact of the noise levels applied to the long-term memory. As demonstrated Tab. 4, when perception conditions are not applied, reducing the noise level in the memory bank improves the overall temporal continuity of the video, mitigating abrupt changes in certain frames. However, due to the gap between the generated video during inference and the ground-truth video during training, accumulated errors exacerbate video drift. In contrast, incorporating perception conditions enhances model robustness, allowing clearer past frames to be input into the model guidance with much lower quality drift. We randomly vary the noise level t. for RGB and flow latents between 0.7 and 0.9 during training to enhance robustness, and set it to a fixed value of 0.8 during inference. Note that higher tm corresponds to lower noise level, consistent with our preliminary.
Table A blation study on noise level. Adding noise to the memory bank reduces drift but reduces consistency. With perceptual conditions, the model is more robust to drift and less sensitive to noise level. The tests are conducted with Cog VideoXo222 on the robotic manipulation dataset.
Subject consistency
Background consistency
Image
Motion
Quality drift
mm
quality
smoothness
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w//
0.9
89.64
91.78
92.17
92.56
0.52
0.60
0.67
0.73
0.19
0.07
0.7
89.23
90.17
91.45
92.37
0.56
0.62
0.69
0.76
0.15
0.06
0.3
88.49
88.98
90.62
91.34
0.60
0.61
0.70
0.75
0.10
0.04
Ablation on number of groups and length of memory bank. To further analyze the impact of memory bank length and the number of prediction groups, we conduct an ablation study on these hyperparameters. Let F' denote the total number of input latent frames, G' the number of prediction groups, N the number of frames per group, and M the memory bank length. These parameters are related by N G + M = F. We fix F 21 for all settings, in order to fix the computational cost. Within the memory bank, one frame is reserved for short-term memory (fully denoised frame), while the remaining M - frames are dedicated to long-term memory. As shown in Tab. 5, considering the overall performance across all metrics as well as the number of frames generated under the same computation, we set M = 5 and N = G = 4 in our final training configuration. This experiment is conducted on the robotic manipulation dataset using 16 100 GPUs, consistent with the ablation studies in the main paper, including the discussion methods for long videos and the analysis of

---

Table 5: Ablation study. Notations such as m1 ___4_______ g4_n5 indicate memory bank length M = 1, prediction groups 4, and frames per group N 5, and so forth, while Frameoutput refers to the number of video frames produced by single complete denoising pass.
Setting
Subject consistency
Background consistency
Image quality
Motion smoothness
Quality drift
frame output
m1 m1_g4_nb
88.85
91.30
0.61
0.75
0.05
20
m5 m5_g4_1 ___4__n4 n4
90.92
92.39
0.60
0.75
0.07
16
m13 [_8_g__n2
91.04
92.51
0.58
0.74
0.09
m5 m6__8__2 ___8_n22 n2
90.64
91.87
0.62
0.73
0.06
16
m1 a1__2_nn0
90.55
91.23
0.56
0.68
0.12
20
noise levels in the memory bank. For these analyses, we extract the first frame from videos where DROID operations fail, using these frames as input images. This subset of data is entirely unseen during training. Based on these first frames, we employ ChatGPT to generate 30 prompts for 15-20 second robotic operations, each containing 3-5 action commands (e.g., move, grasp, switch, push/pull). For the ablation study on the contribution perceptual conditions, we use 60 short 5-second videos isolate the effects from long video frameworks.
Drift resistance experiment details. To investigate the drift resistance of perceptual conditions and color information, we conduct the following experiment after model training. For the depth-only and RGB-only variants, we retain only the weights corresponding to the relevant input and output channels, while loading all other parameters from the fully trained model. We then select a single image or depth map and replicate across frames (to match the model's input sequence length), appending "static" to the training caption. This process generates 10,000 such samples, which we fine-tune for 1,000 steps each, enabling the model to perform a simple task: consistently outputting static images in streaming manner. Notably, since optical flow represents motion between frames and lacks temporal continuity as a signal, its isolated output lacks meaningful consistency, so we exclude it from this experiment. We assess the drift resistance by measuring the normalized Mean Squared Error (MSE) between the subsequently generated frames and the first generated frame. As shown in Fig. 5, depth output alone demonstrates superior drift resistance. Jointly outputting both RGB and depth mitigates the drift phenomenon RGB.
0.05
Depth
dapth
Normaited
0.02
0.01
50
100
150
200
Frame
Figure 5: Normalized mse for drift resistance.
Conclusion
We present WorldWeaver, framework for long video generation that jointly model RGB information and perceptual conditions under unified long-context modeling pipeline. We find that jointly predicting RGB and perceptual signals leads improvements both consistency and motion quality, Leveraging perceptual conditions, especially depth, which is more resistant to drift than color, enables us better preserve historical context and enhance temporal consistency. Furthermore, by adopting non-causal prediction groups and group-wiss wise noise strategy that ensures alignment between training and inference, we are able to alleviate drift and reduce the overall training cost. Extensive experiments on aabotic-manipulattonn and in-the-wild datasets show that WorldWeaver improves long-horizon stability and visual fidelity. achieves superior consistency and drift mitigation, confirming its effectiveness.
10

---

References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadlyay. Yongxin Chen, Yin Cui, Yifan Ding. et al. Cosmos world foundation model platform for physical arXiv preprint ariiv:2501.03555 2025.
[2] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58791, 2024.
[3] Kevin Black, Mitsuhiko Nakamoto. Pranav Aireya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 10639, 2023.
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pages 22556--2277,, 2023.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
[6] Jake Bruce, Michael D Dennis. Ashley Edwards. Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar. Richie Steigerwald, Chris Apps. al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[7] Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. arXiv preprint arXiv:2411.00766 2024.
[8] Chi-Lam Cheang, Guangzeng Chen, Jing. Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yiehu Yang. et al. Gr-2: A generative aaleo-lanuage--cttinn model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158 2024.
[9] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak. Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv:2502..04492 2025.
[10] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz. Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-loken prediction meets full-sequence diffusion. NIPS, 37:24081-24122 2024.
[11] Guibin Chen, Dixuan Lin, Jiangping Yang. Chunze Lin, Junchen Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengcheng Ma, Weiming Xiong, Wei Wang. Nuo Pang. Kang Kang. Zhiheng Xu, Yuzhe Jin, Yupeng Liang, Yubing Song, Peng Zhao, Boyuan Xu, Di Qiu, Debang Li, Zhengcong Fei, Yang Li, and Yahui Zhou. Skyreels-v2: Infinite-length film generative model, 2025. URL https: //arxiv. org/ abb//504. 13074.
[12] Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, al. Goku: Flow based video generative foundation models. arXiv preprint arXiv:2502.0 04896, 2025.
[13] Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. Onc-minute video generation with test-time training. arXiv preprint arXi::2504.05298, 2025.
[14] Exched Decart, Q McInttre, S Campbell, Xinlei Chen, and R Wachen, Oasis: A universe in transformer, 2024.
[15] Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.0355 2024.
[16] Yilun Du, Sherry Yang, Bo Dai. Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. NIPS, 36:91666911999 2023.
[17] Zixun Fang, Wei Zhai, Aimin Su, Hongliang Song, Kai Zhu, Mao Wang, Yu Chen, Zhiheng Liu, Yang Cao. and Jun Zha. Vivid: Video virtual try-on using diffiusion models. arXiv preprint arXiv:2405.1 1199, 2024.
[18] Zixun Fang, Kai Zhu, Zhiheng Liu, Yu Liu, Wei Zhai, Yang Cao, and Zheng Jun Zha. Panoramic video generation with pretrained diffusion models. arXiv preprint arXiv:2506.238: 23513, 2025.
11

---

[19] Ruili Feng, Han Zhang, Zhantao Yang. Jie Xiao, Zhilei Shu. Zhiheng Liu, Andy Zheng, Yukun Huang. Yu Liu, and Hongyang Zhang. The matrix: anfinitehhorizon world generation with real-time moving control. arXiv preprint urX.5:241..03568. 2024.
[20] Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong. Zhenguo Li, and Qiang Xu. High-resolution long video generation for autonomous driving with adaptive control. arXiv preprint urXiv:2411.13807. 2024.
Magicdrivedit:
[21] Shenyuan Gao, Jiazhi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile conirollability. arXiv preprint arXiv:2405.17398 2024.
[22] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1)): 139-144, 2020.
[23] Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.1993 19325, 2025.
[24] Yuwei Guo, Ceyuan Yang. Ziyan Yang, Zhibei Ma. Zhijie Lin, Zhenheng Yang. Dahua Lin, and Lu Jiang. Long context tuning for video generation. arXiv preprint arXiv:2503. 2025.
[25] Gunshi Gupta, Karmesh Yadav, Yarin Gal, Dhruv Batra, Zsolt Kira, Cong Lu, and Tim GJ Rudner. Pre-trained text-to-image diffusion models are versatile representation learners for control. NIPS, 37: 74182-74210, 2024.
[26] David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1rXiv:1803.10122 100,2,022,,,0000 2018.
[27] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1911v:1112 2019.
[28] Danijar Hafner, Timothy Lillicrpp Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[29] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2300 100044,0444 2023.
[30] Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Learning an actionable discrete diffusion policy via large-scale actionless video pre-training. arXiv preprint urXiv:2402.14400 2024.
[31] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent vidco diffusion models for high-fidelity long vidco generation. arXiv preprint 221 2022.
[32] Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streaming(2v; Consistent, dynamic, and extendable long vidco generation from text. arXiv preprint arXiv:2403. 14773, 2024.
[33] Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang. Mohi Khansari, and Yunfei Bai. Retinagan: An object- aware approach to sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pages 10920- 10926. IEEE, 2021.
[34] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
[35] Anthony Hu, Lloyd Russell, Hudson Yco, Zak Murez, George Fedoscev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-l: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 17080, 2023.
[36] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2024.
[37] Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.1355. 2023.
[38] Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang. Nan Zhuang, Quzhe Huang. Yang Song. Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint 45X5542410.05954, 2024.
12

---

[39] Anssi Kanervisto, Dave Bignell, Linda Yilin Wen. Martin Grayson, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao, al. World and human action models towards gameplay ideation. Nature, ( 2025.
[40] Bingyi Kang. Yang Yue. Rui Lu,, Zhijie Lin, Yang Zhao, Kaixin Wang. Gao Huang, and Jiashi Feng. How is video generation from world model: physical law perspective. arXiv preprint arXiv:2411.02385. 2024.
[41] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages 8110 8119, 2020.
[42] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina. Marion Lepert, Yecheng Jason Ma. Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lec, Youngwoon Lee, Marius Memmel, Sungjac Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mereat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong. Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung. Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin. Zehan Ma, Abhiram Maddukuri. Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son. Stephen Tian, Emi Tran, Andrew E. Wang. Yilin Wu, Annie Xie. Jingyun Yang. Patrick Yin. Yunchu Zhang. Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph Lim, Jitendra Malik, Roberto Martin-Martin, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[43] Yaron Lipman, Ricky TQ Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 20. 2022.
[44] Hao Liu, Wilson Yan, Matci Zaharia, and Pieter Abbeel. World model on million-length vidco and language with blockwise ringattention. arXiv preprint arXiv:2402.0 2024.
[45] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint 23:3:22300.05/22 2023.
[46] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects. Advances in neural information processing systems, 36:57500-55519, 2023.
[47] Zhiheng Liu, Ka Leong Cheng, Qiuyu Wang, Shuzhe Wang, Hao Ouyang, Bin Tan, Kai Zhu, Yujun Shen, Qifeng Chen, and Ping Luo. Depthlab: From partial to complete. arXiv preprint 28X1::2412.18153, 2024.
[48] Zhiheng I Liu, Hao Ouyang, Qiuyu Wang. Ka Leong Cheng, Jie Xiao, Kai 7hu, Nan Xue, Yu Liu, Yujun Shen, and Yang Cao. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior. arXiv preprint arXiv:2404.1 1613, 2024.
[49] Zhiheng Liu, Ka Leong Cheng, Xi Chen, Jie Xiao, Hao Ouyang. Kai Zhu, Yu Liu, Yujun Shen, Qifeng Chen, and Ping Luo. Manganinja: Line art colorization with precise reference following. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5666-5677, 2025.
[50] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:
arXiv preprint
2017.
[51] Jiachen Lu, 7e Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. In ECCV, pages 329-345. Springer, 2024.
[52] Yu Lu, Yuanzhi Liang, Linchao Zhu, and Yi Yang. Freelong: Training- free long vidco generation with spectralblend temporal attention. arXiv preprint arXiv:2407. 19918, 2024.
[53] Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He. Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer vidco diffusion via noise rescheduling. arXiv preprint arXiv:2310. 1..::2300.15699 15169, 2023.
[54] Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. RI Rl-cyclegan: Reinforcement learning aware simulation-to rearal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11157- 2020.
13

---

[55] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint Xiv::2008000714, 2024. URL https: //arxiv. ///aaiiv.rg/aaaa/...... wwt//rr/////aeeeeee 007144.
[56] Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer- based world models are happy with 100k interactions. arXiv preprint arXiv:2303.07100 2023.
[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, pages 10684-10695, 2022.
[58] David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. arXiv preprint arXiv:2402.0944777 094700 2024.
[59] Sand-Al. Magi- 1: Autoregressive video generation scale, 2025. URL https////s/t world/ static/filles/MGGI.11 pdd.
[60] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du Russ Tedrake, and Vincent Sitzmann. History- guided video diffusion. arXiv preprini arXiv:2502.06764 2025.
[61] Zhenxiong Tan, Xingyi Yang, Songhua Liu, and Xinchao Wang. Video-infinity; Distributed long video generation. arXiv preprint arXiv:2406. 16260, 2024.
[62] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines, arXiv preprint arXiv:2408.1488 2024.
[63] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint wrXiv:2503.20314. 2025.
[64] Yun Wang. Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li, Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. /8244 2023.
[65] Xiang Wang. Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou Videocomposer: Compositional video synthesis with motion controllability. NIPS, 36:7594-766 2023.
[66] Yihan Wang, Lahav Lipson, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow. arXiv preprini arXiv:2405. 2024.
[67] Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):770280, 1989.
[68] Honghui Yang, Di Huang, Wei Yin, Chunhua Shen, Haifeng Liu, Xiaofei He, Binbin Lin, Wanli Ouyang, and Tong He. Depth any video with scalable synthetic data, arXiv preprint urXiv:2410.10811 2024.
[69] Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al. Generalized predictive model for autonomous driving. In CVPR, pages 14662-14672, 2024.
[70] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. 7:Xxv:2406.09411 2024.
[71] Mingyu Yang, Junyou Li, Zhonghin Fang, Sheng Chen, Yanghin Yu, Qiang Fu, Wei Yang, and Deheng Ye. Playable game generation. arXiv preprint arXiv:24122 12.22242220887.. 2024.
[72] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-vides diftusion models with an expert transformer. arXiv preprint arXiv 06072, 2024.
[73] Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang. Minheng Ni, Zhengyuan Yang. Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303.1 12346, 2023.
[74] Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. arXiv preprint 2rX1v:2412.07772, 2024.
[75] Jiwen Yu, Yiran Qin, Xintao Wang. Pengfei Wan, Di Zhang, and Xihui Liu, Gamefactory: Creating new games with generative interactive videos. arXiv preprint arXiv:2501.0892. 2025.
14

---

[76] Lvmin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for video generation. arXiv preprint arXiv:2504. 12626, 2025.
[77] Deyu Zhou, Quan Sun, Yuang Peng. Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, Xiangyu Zhang, Lionel M Ni, et al. Taming teacher forcing for masked autoregressive video generation. arXiv preprint (rxXi::5501123899 2025.
[78] Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators, arXiv:2406. 12802, 2024.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims we made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the supplemental material.
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledger in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper is not theoretical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Ours results is reproducible, and we will release the codes.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We release the codes and in supplementary materials
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /ttpsscccppu/////////// httpc//pcpppcpcceeeeeeeeeee http///ooeeSuiieeeeeeee iiioinnicyy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
The methodologies employed in this work involve computational models that yield consistent and repeatable outputs without variability under the same conditions. These tests are based on established simulations that deterministically produce the same results each time they run, provided the input parameters remain unchanged.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Sufficient information on the computer resources provided in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http///eerpp...e/ wwtrriis.ccpppuiiii cctppclcc///ccccccccccccccccccct cttiiiiiiieleeeeeeeeee
Answer: [Yes]
Justification: We understand the importance of ethical standards in research and take this matter seriously.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: material.
We discuss the potential societal impacts of our work in the supplemental
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
20

---

Answer: [Yes]
Justification: We commit to continuously monitoring the usage of our released models and codes and will take action restrict access or provide additional guidance if identify concerning patterns of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets used have been properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
21

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method development in this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM policy (https: /nnuriiss.... / for what should or should not be described.
22

---

The content of this supplementary PDF is organized as follows:
Provided more visualizations and comparisons.
Systematically conducted a user study to compare our method with open-source models.
Additional ablation and analysis experiments.
Discussion on limitations and broader impact.
Contents
Introduction
2
Related Works
3
Methods
3.1
Preliminary
3.2 Enhancing Video Generation with Perceptual Conditions
3.3
Long-Horizon Generation Framework
Experiments
4.1
Experimental Setup
4.2
Comparisons
4.3
Ablation studies
5
Conclusion
10
Additional Visual Results
23
6.1
More visualizations
23
User Study
23
7.1
Limitations and future works
25
6 Additional Visual Results
6.1 More visualizations
As shown in Fig. 6, we present additional videos generated by WorldWeaver. Additionally, we recommend accessing the viewer.html file for visual comparison between our approach and current state-of-the-art methods
User Study
To conduct comprehensive comparison, we evaluate our model (based on Wan2.1 1.3B [63]) against recently released state-of-the-ar long-video models of similar scale, including SkyReels-V V2 1.3B [11] and MAGI 4.5B [59]. Specifically, we use 48 prompts, each containing 4-6 actions, to generate videos lasting 20-30 seconds. All prompts are provided in the prompts. txt file. We engage 15 annotators to complete a questionnaire, with the screenshot presented Fig. The questionnaire comprises five questions: (1) Which video has the highest overall image quality? (2) Which video exhibits better consistency (considering both subject and background)? the smallest difference between the first 5 seconds and the last 5 seconds (in terms of quality and
(3) Which video shows
23

---

The two people walk towards each other,
then they have conversation,
MMM they each raise one hand and put them together, and the girl smiles happily.
college student studies in a campus cafe, taking notes his textbook,
checking his laptop, glancing at messages on his phone,
and finally packing up his things when friend arrives.
A young man jogs around peaceful lake at dawn,
he stops to catch his breath and stretch,
he takes photo of the sunrise, then continues running.
Figure 6: More visualizations.
consistency)? (4) Which video has smoother motion? (5) Which video aligns best with the overall action described in the prompt.
Results are shown Fig. Our method demonstrates performance comparable to MAGI in motion smoothness and consistency metrics, while achieving superior results in quality drift. However, slight gap image quality remains, with overall performance generally surpassing SkyReels-V? V2. These findings are generally consistent with our measurements the VBench benchmark.

---

Sart
Please the five questions
Prompt: A bey skips stones by the riverbank, he cheers when one skips three times, he searches for flatter stone, then tries again with determination.
Wesk the wegeettddddttennnddeeooo
Vitel
Vota
Vew
Tie
vidio sonaeeeeeeee artiaaly craditiing contmeryy t. cndtagooneeet and badgpaad appanaer chaghuu
Vote
videogaaigh of cour
widd vito morte madanvehh ná disanden catieeat
Vite
Vote
Vele3
Wish Wioul pu buttvith oveull acdive oee bediiceeeeeeee thapnep?
Figure 7: Visualization of the user study interface.
7.1 Limitations and future works
Limitations and future works. Despite its strengths, our work has several limitations. First, learning stable physical dynamics from the complexity of the real world remains long way off, and model far from perfect. Operations involving very small objects can still exhibit sudden disappearances, since even depth-based cues cannot fully capture these small objects. Second, although fine-tuning existing models allows us to generate 10~200s videos, the success rate of generation still decreases as the video length increases, and eтror accumulation persists longer horizons, which leave as an important direction for future work. Finally, while video depth is shown to be the most effective perceptual condition in our experiments, investigating additional complementary cues further exploit the rich information in real-world data also represents promising avenue for further study. Broader impacts. As video generation technology continues to advance, the development of robust authentication and forgery detection methods becomes increasingly critical. The ability to create highly realistic videos, such as those generated by our model, underscores the need for parallel advancements in counterfeit identification to mitigate potential misuse, ensuring the integrity of digital content in an era of rapid technological evolution.
25

---

Method
Image Quality
16.8%
13.5%
25.6%
44.2%
ours sky-v2 magi
Consistency
28.8%
8.1%
24.9%
38.3%
tie
Drift
33.5%
11.1%
25.1%
30.4%
Motion
15.3%
46.5%
Text
17.2%
24.9%
14.1%
43.6%
Figure 8: Results of user study.
26


Reviews:
[{"review_id": "6bNj8b9PHD", "reviewer": "Reviewer_vMBw", "summary": "This paper proposes WorldWeaver, a framework for long-horizon video generation aimed at addressing the challenges of structural and temporal consistency in generative video modeling. The key contributions include predicting perceptual conditions (depth and optical flow) alongside RGB frames using the same model, implementing a memory bank to preserve context through depth stability, and introducing segmented noise scheduling to mitigate temporal drift. The authors validate their method using both two video models on one general-purpose dataset and one robotic-manipulation dataset, demonstrating improvements over baseline approaches in consistency and drift metrics.", "questions": "1. How robust is the video depth processing pipeline?\n\n2. How will the depth prediction quality affect generative performance?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- This work leverages carefully curated video depth predictions (using Video Depth Anything and Depth Anything V2) alongside optical flow predictions to effectively mitigate temporal drift and structural degradation, issues commonly observed in video generative models.\n\n- It introduces a memory bank strategy that maintains noise-free depth information and slightly noised RGB and optical flow information, enhancing model robustness.\n\n- To further improve upon existing methods like diffusion forcing and history guidance, the paper proposes a segmented noise scheduling strategy, specifically alleviating exposure bias issues in long-sequence video generation.\n\n- Overall, the paper's presentation and clarity are commendable, despite some minor issues.\n\n\n### Weakness\n- Limited novelty. \n    - First of all, I understand this field is crowded and it is hard to study something unexceptedly new or interesting. \n    - But this work mainly primarily builds upon and extends ideas from existing works, particularly VideoJAM, by adding depth as additional input&output. Furthermore, the techniques proposed to handle exposure bias are incremental improvements of known methods like those from diffusion forcing / history-guidance, noisy-context learning works.\n\n- On comparison and ablation studies.\n    - Most experiments, apart from those in Table 1, are conducted exclusively on the robotic manipulation dataset, DROID (I didn't find what dataset Table 3 is using). In this dataset, camera motion is absent and object movements are relatively fixed. The observations and conclusions drawn from robotic datasets hardly transfer to general-purpose datasets. The authors should include more experiments on general-purpose datasets to strengthen their claims.\n    - The definition of \"converged models\" used for comparisons in ablation studies (mentioned in line 259) is unclear. Given that lower loss in flow matching or diffusion training does not necessarily imply better generative performance in many cases, it is essential to allocate equal computational resources to each method to ensure fair comparisons.\n    - In Table 1, it would be highly beneficial to include a baseline that employs the same base model as the proposed method, potentially using a repeated image-to-video extension strategy if direct long-video synthesis is not feasible. This comparison would more clearly indicate how much performance improvement stems directly from the proposed components.\n\nMinors:\n- It would be helpful to provide a more comprehensive preliminary section for a broader audience, such as delayed denoising. \n- Given the critique of VideoJAM’s \"complex sampling-stage guidance,\" it is necessary for the authors to clarify why the guidance approach in VideoJAM is considered complex and how the proposed method simplifies or improves upon it.\n- There're many typoes in intro, e.g., \"strategyassinignig\", \"inferenceto\", perceptual-conditioned \"next prediction\", and more. Please fix them.\n \n\nI'm happy to raise my score.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I believe the paper is above the acceptance bar despite the concern of novelty. However, the messages conveyed by the paper, e.g., depth > optical flow for long video generation, are useful for the community. Thus, I would like to keep my original rating.", "rebuttal": "### **Author Response to Reviewer vMBw**\nWe sincerely appreciate your kind words about our work. We believe the response below clarifies your concern.\n\n **Q1: Limited novelty.**\n\n **A1:**  Thank you for your feedback. While our work is partially inspired by VideoJAM, our contributions are distinct in two key aspects.\n- First, we systematically analyze different perceptual conditions and find that video depth offers more comprehensive improvements —enhancing both motion quality and temporal consistency—compared to optical flow. Its robustness to visual drift is a key insight that motivates our design.\n- Second, we address a dilemma in prior methods like Diffusion Forcing: low noise in historical frames leads to drift, while high noise limits useful context. We resolve this by leveraging depth’s stability and proposing a memory-augmented framework that retains clearer historical cues. This allows effective use of long-term context without sacrificing consistency.\n\n **Q2: Ablation study on general-purpose datasets.**\n\n **A2:** We completely agree with your point that evaluation on general-purpose datasets is essential for validating broader applicability. Due to computational cost constraints, we selected a subset of baselines—CausalVid, Diffusion Forcing, as well as our own framework without perceptual conditioning—and retrained them on the same general-purpose dataset using the Wan-1.3B backbone for direct comparison. These results are consistent with the findings reported in the main paper.\n\n| Methods         | Subject consistency | Background consistency | Image quality | Motion smoothness | Δ Quality drift |\n|-----------------|---------------------|-------------------------|---------------|-------------------|-----------------|\n| CausVid  |  87.06             | 89.68                  | 0.55          |    0.83          |  0.11         |\n| DF  |  86.72                | 89.42                  | 0.56         |    0.86          |  0.07       |\n| Ours *w/o* perceps   |  86.81                | 89.54                  | 0.55         |    0.82          |  0.12         |\n| Ours   |  87.34                 | 90.49                  | 0.59          |    0.87          |  0.07         |\n\n **Q3: Definition of Converged Models.**\n\n **A3:**  Thank you very much for pointing this out. We used equal computational resources and training configurations for all methods to ensure fairness, except for the total number of training steps, which was intentionally varied to investigate the convergence behavior of different algorithms. In practice, we do not rely solely on loss values to determine convergence. Instead, we monitor both quantitative evaluation metrics and visual results at inference time. Once both have stabilized and no further improvement is observed, we consider the model to be converged and record the corresponding number of training steps. We also continue training beyond the observed convergence point to ensure that no further improvements can be achieved. We will clarify this definition explicitly in the revised version.\n\n **Q4: Lack of a fair baseline using same base model.**\n\n **A4:** Thank you for the insightful suggestion. Our method is based on a 1.3B text-to-video (T2V) model, while Wan does not provide a corresponding 1.3B image-to-video model. As an alternative, we use the Wan-14B image-to-video model for evaluation on the same benchmark. It is worth noting that these pretrained model baselines have undergone high-quality supervised fine-tuning (SFT), whereas our training data is not as high-quality as the curated SFT data.\n\nAs shown in the table below, we observe that increasing model size does lead to improvements in overall video quality and motion scores. Nevertheless, these larger models still exhibit more pronounced temporal drift. This is largely due to the repeated image-to-video generation strategy, which lacks access to prior context and suffers from a severe training-inference gap.\n\n\n| Methods         | Subject consistency | Background consistency | Image quality | Motion smoothness | Δ Quality drift |\n|-----------------|---------------------|-------------------------|---------------|-------------------|-----------------|\n| WanI2V 14B    | 89.86               | 91.02                 | 0.63          |       0.90        | **0.15**          |\n| Ours 1.3B    |  87.34                 | 90.49                  | 0.59          |    0.87          |  0.07         |\n\nDespite using a smaller 1.3B model, our method significantly mitigates temporal drift and improves long-horizon video quality through its architectural and training design. Moreover, as presented in Table 2 of the main paper, we compare repeated image-to-video generation with other autoregressive and streaming baselines on the same dataset, and consistently find that autoregressive generation is more suitable for long-horizon video synthesis.\n\n\n **Q5: Clarification.**\n\n  **A5.1 Preliminary:** Thank you for the suggestion. We will expand the preliminary section in the revised version to better support a broader audience\n\n **A5.2 Complexity of VideoJAM's sampling guidance:** VideoJAM’s sampling guidance is considered complex for two main reasons. First, it requires additional forward passes during inference, which increases computational overhead. Second, it introduces multiple guidance scales as additional inference-time parameters. These parameters can affect generation quality and may require manual adjustment depending on the scene or prompt.\n\n**A5.3 Typos:** We will carefully proofread the paper and correct all typographical and formatting errors in the revised version.\n\n**A5.4 Depth estimation robustness:** Our depth pipeline is based on state-of-the-art models (Video Depth Anything), which are robust across a wide range of scenes, making the overall depth processing pipeline reliable in practice. Crucially, our method does not rely on precise metric depth. We apply an affine‑invariant normalization to convert raw depth maps into relative depths in \\([-1,1]\\), following Marigold[1]:\n\n$$\n\\tilde{d} = \\left(\\frac{d - d_{2}}{d_{98} - d_{2}} - 0.5\\right)\\times 2\n$$\n\nwhere $ d_{2} $ and  $ d_{98} $ are the 2nd and 98th percentiles of each frame’s depth distribution. This removes extreme outliers and global scale/shift variations so the model focuses on relative spatial relationships. Depth serves as a relative structural prior, providing stable geometric cues that help maintain temporal consistency and stabilize long-horizon generation—which current models are sufficient to support.\n\n[1] Ke, Bingxin, et al. \"Repurposing diffusion-based image generators for monocular depth estimation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024."}, {"review_id": "Ui3HNQpSFS", "reviewer": "Reviewer_bGie", "summary": "The paper WorldWeaver targets the drifting (error accumulation) and gradual structure‑deformation problems of autoregressive generation models. The authors propose to use multi‑modal prediction, including RGB, monocular depth (VideoDepth‑Anything) and optical flow (SEA‑RAFT) (all encoded by a 3D VAE and concatenated). The DiT/Flow‑Matching backbone is fine‑tuned to regress jointly for all these channels. It also uses a perceptual memory bank, which is a sliding window of past frames is split into short‑term (noise‑free) and long‑term memories; the RGB+flow part of long‑term memory is mildly noised while depth remains clean, under the hypothesis that geometry is more drift‑resistant. The authors also propose to do segmented group‑wise noise schedule aligned between training and inference; groups are denoised sequentially with delayed denoising. Significant improvement can be observed from the generated videos comparing with Magi-1 and SkyReels-v2, two SoTA autoregressive video generation models.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n-- The motivation is clear: RGB‑only prediction has bias and this is a good motivation for multi‑perception conditioning, which is supported by toy “static scene drift” experiment (Fig. 3).\n\n-- The authors provide comprehensive different types of multi-modal inputs and outputs, which are thoroughly considered.\n\n-- The multi-modal memory‑bank design grounded in empirical observation that depth drifts slower; elegant to keep depth clean and RGB slightly noised.\n\n-- Comprehensive ablations are provided on perception types, memory‑bank noise, and training noise schedule.\n\n\nWeakness:\n\n-- The idea of including multi-modality inputs and outputs are relatively straightforward and has been discussed (e.g. VideoJAM at ICML'25), and its \"memory bank\" is more of a diffusion forcing variant when dealing with more than RGB channels. The novelty is arguably a bit incremental but I think the authors executed everything well.", "was_revised": true, "final_justification": "I do not have much problem with this paper initially and would keep my score.", "rebuttal": "### **Author Response to Reviewer bGie**\nWe sincerely appreciate your valuable feedback and your positive assessment of our work. Below, we address your comments in detail.\n\n **Q1: Clarifying the contributions beyond existing methods.**\n\n **A1:** While our work is partially inspired by VideoJAM, our contributions are distinct in two key aspects.\n- First, we systematically analyze different perceptual conditions and find that video depth offers more comprehensive improvements —enhancing both motion quality and temporal consistency—compared to optical flow. Its robustness to visual drift is a key insight that motivates our design.\n- Second, we address a fundamental tension in prior methods like Diffusion Forcing: low noise in historical frames leads to drift, while high noise limits useful context. We resolve this by leveraging depth’s stability and proposing a memory-augmented framework that retains clearer historical cues. This allows effective use of long-term context without sacrificing consistency."}, {"review_id": "3YFOoBzDNl", "reviewer": "Reviewer_6nHu", "summary": "This work tackles the task of long video generation, with three contributions: 1) a joint modeling framework that predicts RGB, depth and optical flow; a perceptual memory bank with depth to record historical frames. 3) A new segmented noise scheduling strategy that reduces the training time. Extensive experiments show improved subject / background consistency and motion smoothness on general-purpose and robotic benchmarks.", "questions": "- The depth and optical flow signals for in-the-wild videos are predicted using off-the-shelf models, which may introduce unaligned signal pairs, inconsistent depth signals and hence accumulated errors. Is there a mechanism to correct or mitigate these errors?\n- What's the benefit of training with the internal general-purpose dataset? It would be good to see results that train with publically available general-purpose datasets for more thorough evaluation.\n- Evaluation-wise, the proposed method is only tested on generated images and texts. How is the performance on in-the-wild test inputs?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Introducing auxiliary perceptual signals depth and optical flow in the long video generation framework is new and proves to maintain consistent motion across long context.\n+ Ablative experiments prove the reduced training epochs and the effectiveness of the perceptual memory bank and the group-wise\n noise scheduling.\n \nWeakness:\n- Paper organization looks a bit off. Section 3.3 is better to be separated into two subsections for each contribution.\n- The depth and optical flow signals for in-the-wild videos are predicted using off-the-shelf models, which may introduce unaligned signal pairs, inconsistent depth signals and hence accumulated errors. Is there a mechanism to correct or mitigate these errors?\n- What's the benefit of training with the internal general-purpose dataset? It would be good to see results that train with publically available general-purpose datasets for more thorough evaluation.\n- Evaluation-wise, the proposed method is only tested on generated images and texts. How is the performance on in-the-wild test inputs?\n- There are mainly static illustrations and only three side-by-side qualitative video comparisons in the paper. It would be better to show more dynamic cases to evaluate the motion quality and temporal consistency.\n- The presented results still contain temporal inconsistency at semantic transitions (supp html case 1, the woman's expression changes from none to laughing is unnatural).", "was_revised": true, "final_justification": "The authors' response addresses most of my concerns. Thus, I tend to keep my original score and lean towards acceptance.", "rebuttal": "### **Author Response to Reviewer 6nHu**\nWe sincerely appreciate your recognition and thoughtful feedback, which are instrumental in refining and strengthening our work. We hope the following response adequately addresses your question.\n\n **Q1: Clarifying the paper organization.**\n\n **A1:** Thank you for pointing this out. In the revision we will split Section 3.3 into two distinct subsections:  (1) Perceptual Memory Bank (design, motivation and ablations), and (2) Group‑wise Noise Scheduling (training/inference alignment, efficiency). This re‑structure cleanly separates the two contributions and improves readability without altering technical content.\n\n **Q2: Depth/Flow mis‑alignment mitigation.**\n\n**A2:** Thank you for the question. For depth we apply an affine‑invariant normalization to convert raw depth maps into relative depths in \\([-1,1]\\), following Marigold[1]:\n\n$$\n\\tilde{d} = \\left(\\frac{d - d_{2}}{d_{98} - d_{2}} - 0.5\\right)\\times 2\n$$\n\nwhere $ d_{2} $ and  $ d_{98} $ are the 2nd and 98th percentiles of each frame’s depth distribution. This removes extreme outliers and global scale/shift variations so the model focuses on relative spatial relationships.\n\nFor optical flow we divide each displacement vector by the frame diagonal, $\\sqrt{H^2 + W^2}$. Since all training clips use the same resolution, this aligns motion magnitudes across the dataset. The normalized flow is then encoded as an RGB video (hue = direction, opacity = magnitude).\n\n **Q3: Benefits of internal datasets.**\n\n**A3:** Our task requires full-model fine-tuning rather than freezing the backbone or applying lightweight adaptation methods such as LoRA. However, most large-scale publicly available datasets contain various quality issues—such as embedded subtitles, watermarks, or overly chaotic motion patterns—which significantly affect the training process. Notably, existing models like the pretrained Wan, Magi, and SkyReels-V2 have all undergone a final-stage supervised fine-tuning (SFT) on carefully curated high-quality data. If we were to further fine-tune such models on noisy public data, it would significantly degrade video quality and obscure the true contribution of our modeling framework. In contrast, our internal general-purpose dataset provides relatively higher data quality—with better control over subtitles, watermark removal. \n\nWe fully agree that more thorough evaluation across diverse datasets is important. As presented in Table 2 of the main paper, we have already conducted a comparison across multiple methods using the same public robotic manipulation dataset. However, due to the inherent difficulty of reliably cleaning and filtering large-scale public video datasets—e.g., removing subtitles, watermarks, and abrupt transitions—we adopted a controlled setting using our internal general-purpose dataset. Specifically, we re-trained multiple baseline methods on the same internal dataset to ensure fair and consistent evaluation across architectures. This setup allows us to isolate the effect of model design without the confounding influence of data quality variation. We will also include additional results on public general-purpose datasets in the revised version to further demonstrate the robustness and generalization ability of our method.\n\n| Methods         | Subject consistency | Background consistency | Image quality | Motion smoothness | Δ Quality drift |\n|-----------------|---------------------|-------------------------|---------------|-------------------|-----------------|\n| CausVid  |  87.06             | 89.68                  | 0.55          |    0.83          |  0.11         |\n| DF  |  86.72                | 89.42                  | 0.56         |    0.86          |  0.07       |\n| Ours *w/o* perceps   |  86.81                | 89.54                  | 0.55         |    0.83          |  0.12         |\n| Ours   |  87.34                 | 90.49                  | 0.59          |    0.87          |  0.07         |\n\n **Q4: Test on  in-the-wild inputs.**\n\n**A4:** We thank the reviewer for their valuable feedback regarding the evaluation on in-the-wild inputs. In our experimental design, we utilized text prompts and initial images generated by a third-party tool (ChatGPT). This approach was intentionally chosen to establish a fair and standardized benchmark for all compared methods, thereby eliminating potential biases that could arise from manually curated prompts.\n\nRegarding the model's performance on in-the-wild data, our method exhibits strong robustness. This is because the model itself was trained on large-scale, in-the-wild datasets. Furthermore, our qualitative results on the robotic manipulation dataset are  generated from real-world scenarios. \n\nWe will incorporate extensive results on in-the-wild videos in the final revision to demonstrate our framework's robustness, as we are limited from showing new results during the rebuttal period.\n\n **Q5: Dynamic cases.**\n\n**A5:** We agree with the reviewer on the importance of showcasing more dynamic cases, particularly those with complex motions and semantic transitions, and will incorporate them into the final version. We candidly acknowledge that while our joint training approach improves upon RGB-only models, generating long videos with large motion (like long dance sequences) remains a common challenge for the field. This is largely due to error accumulation in streaming frameworks, where a single degraded frame can impact the entire subsequent sequence.\n\nOur method helps to mitigate this issue. By jointly modeling RGB with stable perceptual cues like depth and flow, it generates a more reliable initial context, thus providing a more robust basis for subsequent frames.\n\n[1] Ke, Bingxin, et al. \"Repurposing diffusion-based image generators for monocular depth estimation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024."}]

Author rebuttal:

Meta-review: This paper addresses long-horizon video generation by jointly predicting auxiliary signals (depth and optical flow) alongside RGB, and introduces a perception-conditioned framework with a memory bank and group-wise noise scheduling to preserve context and reduce drift.

The paper initially received three borderline accepts. Reviewers highlighted several strengths:
- Clear motivation for introducing auxiliary signals;
- Comprehensive consideration of multi-modal inputs and outputs;
- Effectiveness of the memory bank and proposed noise scheduling.

Concerns were raised about novelty, baselines, ablation, and the use of an internal dataset. While most issues were addressed in the rebuttal, novelty concerns remain, as multi-modal generation has been explored in prior works (e.g., VideoJAM), and the segmented noise scheduling extends Diffusion Forcing. Nevertheless, the paper is well-executed and conveys useful insights, and all reviewers retained their borderline-accept scores. The AC considers the merits sufficient to warrant acceptance.
Decision: Accept (poster)