Title: WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception
Venue: NeurIPS 2025
Accepted: yes
Keywords: long video generation

world models
Abstract: Generative video modeling has made significant strides, yet ensuring structural and temporal consistency over long sequences remains a challenge. Current methods predominantly rely on RGB signals, leading to accumulated errors in object structure and motion over extended durations. To address these issues, we introduce WorldWeaver, a robust framework for long video generation that jointly models RGB frames and perceptual conditions within a unified long-horizon modeling scheme. Our training framework offers three key advantages. First, by jointly predicting perceptual conditions and color information from a unified representation, it significantly enhances temporal consistency and motion dynamics. Second, by leveraging depth cues, which we observe to be more resistant to drift than RGB, we construct a memory bank that preserves clearer contextual information, improving quality in long-horizon video generation. Third, we employ segmented noise scheduling for training prediction groups, which further mitigates drift and reduces computational cost. Extensive experiments on both diffusion and rectified flow-based models demonstrate the effectiveness of WorldWeaver in reducing temporal drift and improving the fidelity of generated videos.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the task of long video generation, with three contributions: 1) a joint modeling framework that predicts RGB, depth and optical flow; a perceptual memory bank with depth to record historical frames. 3) A new segmented noise scheduling strategy that reduces the training time. Extensive experiments show improved subject / background consistency and motion smoothness on general-purpose and robotic benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Introducing auxiliary perceptual signals depth and optical flow in the long video generation framework is new and proves to maintain consistent motion across long context.\n+ Ablative experiments prove the reduced training epochs and the effectiveness of the perceptual memory bank and the group-wise\n noise scheduling.\n \nWeakness:\n- Paper organization looks a bit off. Section 3.3 is better to be separated into two subsections for each contribution.\n- The depth and optical flow signals for in-the-wild videos are predicted using off-the-shelf models, which may introduce unaligned signal pairs, inconsistent depth signals and hence accumulated errors. Is there a mechanism to correct or mitigate these errors?\n- What's the benefit of training with the internal general-purpose dataset? It would be good to see results that train with publically available general-purpose datasets for more thorough evaluation.\n- Evaluation-wise, the proposed method is only tested on generated images and texts. How is the performance on in-the-wild test inputs?\n- There are mainly static illustrations and only three side-by-side qualitative video comparisons in the paper. It would be better to show more dynamic cases to evaluate the motion quality and temporal consistency.\n- The presented results still contain temporal inconsistency at semantic transitions (supp html case 1, the woman's expression changes from none to laughing is unnatural).\n\nQUESTIONS:\n- The depth and optical flow signals for in-the-wild videos are predicted using off-the-shelf models, which may introduce unaligned signal pairs, inconsistent depth signals and hence accumulated errors. Is there a mechanism to correct or mitigate these errors?\n- What's the benefit of training with the internal general-purpose dataset? It would be good to see results that train with publically available general-purpose datasets for more thorough evaluation.\n- Evaluation-wise, the proposed method is only tested on generated images and texts. How is the performance on in-the-wild test inputs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper WorldWeaver targets the drifting (error accumulation) and gradual structure‑deformation problems of autoregressive generation models. The authors propose to use multi‑modal prediction, including RGB, monocular depth (VideoDepth‑Anything) and optical flow (SEA‑RAFT) (all encoded by a 3D VAE and concatenated). The DiT/Flow‑Matching backbone is fine‑tuned to regress jointly for all these channels. It also uses a perceptual memory bank, which is a sliding window of past frames is split into short‑term (noise‑free) and long‑term memories; the RGB+flow part of long‑term memory is mildly noised while depth remains clean, under the hypothesis that geometry is more drift‑resistant. The authors also propose to do segmented group‑wise noise schedule aligned between training and inference; groups are denoised sequentially with delayed denoising. Significant improvement can be observed from the generated videos comparing with Magi-1 and SkyReels-v2, two SoTA autoregressive video generation models.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n-- The motivation is clear: RGB‑only prediction has bias and this is a good motivation for multi‑perception conditioning, which is supported by toy “static scene drift” experiment (Fig. 3).\n\n-- The authors provide comprehensive different types of multi-modal inputs and outputs, which are thoroughly considered.\n\n-- The multi-modal memory‑bank design grounded in empirical observation that depth drifts slower; elegant to keep depth clean and RGB slightly noised.\n\n-- Comprehensive ablations are provided on perception types, memory‑bank noise, and training noise schedule.\n\n\nWeakness:\n\n-- The idea of including multi-modality inputs and outputs are relatively straightforward and has been discussed (e.g. VideoJAM at ICML'25), and its \"memory bank\" is more of a diffusion forcing variant when dealing with more than RGB channels. The novelty is arguably a bit incremental but I think the authors executed everything well.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes WorldWeaver, a framework for long-horizon video generation aimed at addressing the challenges of structural and temporal consistency in generative video modeling. The key contributions include predicting perceptual conditions (depth and optical flow) alongside RGB frames using the same model, implementing a memory bank to preserve context through depth stability, and introducing segmented noise scheduling to mitigate temporal drift. The authors validate their method using both two video models on one general-purpose dataset and one robotic-manipulation dataset, demonstrating improvements over baseline approaches in consistency and drift metrics.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- This work leverages carefully curated video depth predictions (using Video Depth Anything and Depth Anything V2) alongside optical flow predictions to effectively mitigate temporal drift and structural degradation, issues commonly observed in video generative models.\n\n- It introduces a memory bank strategy that maintains noise-free depth information and slightly noised RGB and optical flow information, enhancing model robustness.\n\n- To further improve upon existing methods like diffusion forcing and history guidance, the paper proposes a segmented noise scheduling strategy, specifically alleviating exposure bias issues in long-sequence video generation.\n\n- Overall, the paper's presentation and clarity are commendable, despite some minor issues.\n\n\n### Weakness\n- Limited novelty. \n    - First of all, I understand this field is crowded and it is hard to study something unexceptedly new or interesting. \n    - But this work mainly primarily builds upon and extends ideas from existing works, particularly VideoJAM, by adding depth as additional input&output. Furthermore, the techniques proposed to handle exposure bias are incremental improvements of known methods like those from diffusion forcing / history-guidance, noisy-context learning works.\n\n- On comparison and ablation studies.\n    - Most experiments, apart from those in Table 1, are conducted exclusively on the robotic manipulation dataset, DROID (I didn't find what dataset Table 3 is using). In this dataset, camera motion is absent and object movements are relatively fixed. The observations and conclusions drawn from robotic datasets hardly transfer to general-purpose datasets. The authors should include more experiments on general-purpose datasets to strengthen their claims.\n    - The definition of \"converged models\" used for comparisons in ablation studies (mentioned in line 259) is unclear. Given that lower loss in flow matching or diffusion training does not necessarily imply better generative performance in many cases, it is essential to allocate equal computational resources to each method to ensure fair comparisons.\n    - In Table 1, it would be highly beneficial to include a baseline that employs the same base model as the proposed method, potentially using a repeated image-to-video extension strategy if direct long-video synthesis is not feasible. This comparison would more clearly indicate how much performance improvement stems directly from the proposed components.\n\nMinors:\n- It would be helpful to provide a more comprehensive preliminary section for a broader audience, such as delayed denoising. \n- Given the critique of VideoJAM’s \"complex sampling-stage guidance,\" it is necessary for the authors to clarify why the guidance approach in VideoJAM is considered complex and how the proposed method simplifies or improves upon it.\n- There're many typoes in intro, e.g., \"strategyassinignig\", \"inferenceto\", perceptual-conditioned \"next prediction\", and more. Please fix them.\n \n\nI'm happy to raise my score.\n\nQUESTIONS:\n1. How robust is the video depth processing pipeline?\n\n2. How will the depth prediction quality affect generative performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}