Title: Frame-Level Captions for Long Video Generation with Complex Multi Scenes
Venue: NeurIPS 2025
Accepted: no
Keywords: Long Video Generation; Video Datasets Caption
Abstract: Generating long videos that can show complex stories, like movie scenes from scripts, has great promise and offers much more than short clips. However, current methods that use autoregression with diffusion models often struggle because their step-by-step process naturally leads to a serious error accumulation (drift). Also, many existing ways to make long videos focus on single, continuous scenes, making them less useful for stories with many events and changes. This paper introduces a new approach to solve these problems. First, we propose a novel way to annotate datasets at the \textbf{frame-level}, providing detailed text guidance needed for making complex, multi-scene long videos. This detailed guidance works with a \textbf{Frame-Level Attention Mechanism} to make sure text and video match closely. For creating the video (inference), we develop \textbf{Parallel Multi-Window Denoising (PMWD)}, a new method that handles a long video as multiple overlapping windows. These windows are processed at the same time (in parallel), and the data in overlapping areas is averaged, which allows information to flow both ways and greatly reduces the error accumulation. A key feature is that each part (frame) within these windows can be guided by its own distinct text prompt. Our training uses \textbf{Diffusion Forcing} to give the model the ability to handle time flexibly, which is needed for these advanced generation methods. We tested our approach on difficult VBench 2.0 benchmarks ("Complex Plots" and "Complex Landscapes") using the WanX2.1-T2V-1.3B model. The results show our method is better at following instructions in complex, changing scenes and creates high-quality long videos. We plan to share our dataset annotation methods and trained models with the research community.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenges of semantic confusion and error accumulation in generating long videos. The authors propose a framework that begins with creating a dataset with dense, frame-level text annotations. To leverage this data, they introduce a Frame-Level Attention mechanism for precise text-to-video alignment and use a Diffusion Forcing training strategy to give the model temporal flexibility. The non-sequential inference method, Parallel Multi-Window Denoising (PMWD), processes the entire video in parallel, with overlapping windows to prevent error buildup.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The authors tackle the challenge of long video generation from a novel perspective, introducing a fresh line of thought for the research community.\n2. The figures and tables are clear and effectively aid the reader in understanding the methodology and results.\n\n**Weaknesses**\n1. The technical contribution is limited. The dataset annotations are derived by prompting an existing MLLM, and the diffusion forcing strategy is directly borrowed from prior work [1]. Additionally, the proposed PMWD method is identical to temporal MultiDiffusion [2], which is a widely adopted strategy in previous long video generation studies [3, 4].\n2. The paper's presentation is confusing. The problem setting (e.g., whether it focuses on single-shot long video generation or multi-shot video generation) is not clearly discussed. The comparison settings in Figures 3 and 4 are also unclear.\n3. The paper lacks any ablation experiments to validate its design.\n\n[1] Diffusion forcing: Next-token prediction meets full-sequence diffusion\n\n[2] Multidiffusion: Fusing diffusion paths for controlled image generation\n\n[3] Avid: Any-length video inpainting with diffusion model\n\n[4] Gen-l-video: Multi-text to long video generation via temporal co-denoising\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provide frame-level annotated dataset and corresponding long-video generation models, together with diffusion forcing and PMWD strategy for long video generation. It overcomes shared deficiency of current baseline models including temporal accumulated errors and struggling of complex scenes, and limited handling of timing. The results shows better quality on Vbench (\"Complex Plots\" and \"Complex Landscapes\"). Great job.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The proposed frame-level video dataset will contribute a lot to the community. It is first-of-a-kind in this large scale.\n2. Frame-level attention, and diffusion forcing helps the consistency and necessary dynamics of complex plot and scene applications.\n3. PMWD introduce a balanced strategy over the coherence for long video generation.\n\nWeakness\n\n1. The paper shows plausible quality on complex long video generation with dynamic scenes. But it is still necessary to include the base performance of general T2V performance by only global text prompt on the entire Vbench benchmark. How does the introduction of frame-level dataset & attention influence the foundational ability of the original backbone model (Wan2.1)?\n\n2. The quantitative evaluation is lack of competitors. It is necessary to include several representative open/close-source model into the comparison (HunyuanVideo, Cogvideo, Kling, Pika, Veo3, etc.).\n\n3. The therotical/practical insight of comparing frame-level attention VS video-level attention is suggested. What does this new fine-grained dataset and approach modify the pretained representation/distribution of the original backbone model? For example, the comparison of the token-level attention mapping and according discussion will be very helpful.\n\n4. The dataset is mainly annotated by Gemini. But the quality of the annotation itself is lack of manual validation. How good the quality of the dataset is crucial. Even a portion of evaluation is suggested.\n\nQUESTIONS:\nSee weakness part, total 4 questions and detailed actions are provided.\n\nAlso, the release of the dataset significantly influences the broad contribution of this paper. The community is looking at you :-)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pipeline for multi-scene long video generation, including data annotation, model design and inference strategy. For data, a 700k video dataset with frame-level captioning is provided. For method, it modifies the cross attention layer of WanX2.1 model, allowing each latent exclusively attends to the paired frame-level caption, enabling context change in long videos. The Diffusion Forcing strategy is utilized in training stage. At inference, instead of using auto-regressive generation, this paper proposes to simultaneously denoise multiple overlapping time windows in parallel to eliminate error accumulation. Ablation studies are conducted on VBench 2.0 to validate the effectiveness of the proposed pipeline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed multi-prompt + PMWD inference strategy is simple yet effective, exhibiting superior quantitative and qualitative performance on complex video generation compared to the autoregressive variants.\n\n2. A video dataset with frame-level dense captioning is provided, which is a valuable contribution to the community.\n\n\nWeakness:\n\n1. The paper is not well-structured and is confusing or unclear in several parts. For example, in Figures 3–4 of the manuscript, the inference strategies used in each row are not indicated. Is the first row using FIFO and the second using PMWD, similar to Figure 4-6 in the appendix? Additionally, none of the figures are referenced or discussed in the main text.\n\n2. No baseline comparison experiments are conducted beyond the ablation variants. The paper does mention works with similar tasks, such as LCT, Mask2DiT, but none of them are qualitatively or quantitatively compared.\n\n3. The annotation pipeline described in the manuscript contradicts the prompt provided in the appendix. Specifically, the claimed strategy of \"automatically choosing between shared or independent descriptions based on the degree of visual change\" in \"Adaptive frame-level annotation\" never occurs in the provided Gemini prompt. The actual annotation strategy appears to be simply sampling frames evenly and annotating each frame with Gemini, which is not adaptive at all.\n\n4. The choice of Diffusion Forcing training strategy is confusing. DF is designed to handle varying noise levels within a sequence to enable autoregressive inference, but since the proposed PMWD inference strategy only denoises tokens at the same noise level per step, I do not see the rationale for using DF training strategy.\n\n5. As suggested in the supplementary videos, the PMWD strategy seems to cause jittering at the boundaries between time windows.\n\nQUESTIONS:\n1. Comparison with baseline long video generation methods should be included.\n\n2. What is the actual annotation pipeline? How is \"adaptive\" frame-level annotation performed? Please give further explanation.\n\n3. Why do you use DF training when the inference only handles unified noise level per step?\n\n4. Why are there temporal jittering in the generation results? Is it a drawback of the PMWD inference strategy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed paper is oriented on solving the challenge of generating long, multi-scene videos by text prompts, where existing autoregressive diffusion models suffer from error accumulation and limited scene diversity. The authors propose three key innovations: first — a frame-level dataset annotation methodology that assigns detailed text descriptions to individual frames or latents using multimodal LLMs, enabling precise video-text alignment, second — a frame-level cross-attention mechanism that binds each video segment’s visual features to its specific text prompt during training, improving semantic fidelity and third — a parallel Multi-Window Denoising inference strategy that processes overlapping video frames in parallel, evaluating average predictions in overlapping regions to enable bidirectional context flow and eliminate sequential error drift. The approach is based at the top of the WanX2.1-T2V-1.3B model and further evaluated on VBench 2.0’s Complex Plots and Complex Landscapes benchmarks. The reported results show significant improvements in prompt adherence, temporal consistency, and reduced semantic confusion  in comparison to global-prompt baselines and sequential methods like FIFO-Diffusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n\nThe paper is technically rigorous. Experiments are comprehensive, using challenging Complex Plots and Landscapes VBench 2.0 categories and introducing a novel metric Confusion Degree to evaluate semantic drift. The results show statistically significant gains in text and video alignment and reduced confusion degree in comparison with the baselines. The proposed approach combines diffusion’s high-quality generation with auto-regressive models for sequence extension and as a result ensures smooth scene transitions, extended video lengths, and maintains both visual richness and temporal consistency.\n\nWeaknesses.\n\nPMWD in terms of parallel processing of multiple windows may lead to high memory/FLOPs overhead, which seems to be underexplored in this paper. If we speak about long videos (1+ minutes and more) there should be some architectural scalability limits which were not properly discussed and may lead to a limited use. At the experimental stage the authors focus on two challenging VBench categories, whereas testing on broader domains (e.g., human actions, DIY operations) would strengthen the claims. The frame-level annotation pipeline relies on Gemini Pro 2.5, so potential biases or failures in prompt generation are not discussed. There description of hyperparameters for Diffusion Forcing, PMWD window size/overlap ratios are not provided in the paper.\n\nClarity. The text is well-structured and mostly clear. The figures provide a good visualization of error accumulation and main improvements. Thekey algorithms (PMWD and Diffusion Forcing) are clearly motivated and described.\n\nSignificance. The proposed approach addresses a critical gap in long video generation, handling multi-scene scenarios with precise control along the temporal axis. The frame-level paradigm could influence video-language modeling beyond diffusion.\n\nOriginality. PMWD is a novel inference pipeline that avoids error accumulation via parallel bidirectional window denoising. The attention at frame-level and Diffusion Forcing provide new solutions to temporal misalignment in multi-scene video generation.\n\nQUESTIONS:\nThe following actions may improve the paper\n1. Computational cost evaluation experiments\n2. Provide clear description of the strategy to chose reference segment in DiffusionForcing and other implementation details, especially when you mention to assign 'cleaner' (earlier) timesteps to past segments and 'noisier' (later) to future segments.\n3. Why only Gemini Pro 2.5 was used for prompt generation. How will the quality metrics value change with other VLM use?\n4. Is there any human interpretation of Confusion Degree?\n5. Can we use PMWD for non-overlapping frame shifts? What will be the effect of such operations?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}