Title: FIFO-Diffusion: Generating Infinite Videos from Text without Training
Venue: NeurIPS 2024
Accepted: yes
Keywords: generative models

diffusion

long video generation

tuning-free
Abstract: We propose a novel inference technique based on a pretrained diffusion model for text-conditional video generation. Our approach, called FIFO-Diffusion, is conceptually capable of generating infinitely long videos without additional training. This is achieved by iteratively performing diagonal denoising, which simultaneously processes a series of consecutive frames with increasing noise levels in a queue; our method dequeues a fully denoised frame at the head while enqueuing a new random noise frame at the tail. However, diagonal denoising is a double-edged sword as the frames near the tail can take advantage of cleaner frames by forward reference but such a strategy induces the discrepancy between training and inference. Hence, we introduce latent partitioning to reduce the training-inference gap and lookahead denoising to leverage the benefit of forward referencing. Practically, FIFO-Diffusion consumes a constant amount of memory regardless of the target video length given a baseline model, while well-suited for parallel inference on multiple GPUs. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines. Generated video examples and source codes are available at our project page.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a training-free diagonal denoising method aimed at extending video diffusion models, which typically generate only short clips, to the production of long videos. The authors have validated their method across multiple baselines and demonstrated that it can be extended to different VDM architectures.\n\nSTRENGTHS:\n- The motivation for this work is commendable; generating long videos is a worthwhile topic to research, as past VDMs could only produce short videos of a few seconds.\n- Viewing denoising long videos as a FIFO-Queue is interesting. From the paper, it appears to work to some extent.\n- The discussion of related works is comprehensive. The overall description of the method is understandable and should be reproducible with some experience in video diffusion.\n\nWEAKNESSES:\n- What do the authors think about training models for longer video generation? SORA can generate videos up to 1 minute with remarkable temporal and even 3D consistency. An important rule in deep learning is scaling up. Methods that still work well or are meaningful when computation resources are upgraded are most valuable. The currently available video generation models are still not satisfactory. Applying them for training-free applications seems only to make the results worse and unsatisfactory.\n- Although the videos presented in the supplementary are long compared to current video generation models, most of them are not attractive enough. In most cases, there are just random motion alterations without any order or controllability. In this way, it seems not enough for practical usage. The most important reason is that you never really have a global perception of the generation process, and therefore the motions and contents tend to be repeated or meaningless alterations. I believe the authors should try to add additional controllability to their method. Otherwise, it is going to be not usable.\n\nQUESTIONS:\nThe method presented in this article is straightforward and provides some insights. The biggest concern might be that the actual results are very poor, with so-called long videos consisting mostly of simple, repetitive actions. At this stage, I can only give a borderline reject score, while looking forward to the opinions of other reviewers.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce FIFO-diffusion, a method to enable pre-trained video diffusion models to generate infinitely-long videos via the proposed diagonal denoising. In diagonal denoising, in contrast to the conventional denoising, the noise levels of the frames in a window form a decreasing sequence. This way, at each denoising step, the clean frame at the head of the queue can de dequeued and a new noisy frame can be enqueued, allowing for generating sequences longer than the ones the model was trained on. The authors additionally propose two techniques to further improve the quality of the generated videos:\n1) Latent partitioning to increase the size of the queue and hence decrease the training-inference gap of diagonal denoising.\n2) Lookahead denoising to allow the denoising process to benefit from the difference in the noise levels of the denoised latents and let the more noisy latents to leverage the information in the less noisy latents to reduce the prediction error.\n\nSTRENGTHS:\nThe paper considers a very interesting and challenging problem of enabling video generation models to generate long videos. The proposed method is simple enough to be directly applied to a variety of video generation models. The paper is well-written, all the components are well-motivated and supported by theory and experiments. The quality of the generated videos is good.\n\nWEAKNESSES:\n1) **Prior work**: Diagonal denoising has been first proposed in [1] for motion generation and in [2] for video generation. In both works the models are trained with the different noise levels support. Discussion with respect to these works is missing in the paper. I understand that by the submission time those works were only published on arxiv, but a discussion would facilitate a proper contextualization of the proposed method within the field.\n\n2) **Limitations**: The main limitation of the method, as discussed by the authors, is the training/inference gap. And the authors state that training with the support of different noise levels could improve the performance. However, if the gap didn't exist (given that there are models that train with different noise levels) the solutions that form the main contribution of the paper would not be needed. Hence, such a statement weakens the contributions.\n\n3) **Evaluation**: Quantitative comparisons are limited. The only quantitative comparison (except for human evaluation) is done through FVD and shows that the prior work (FreeNoise) performs better than the proposed method. Moreover, the FVD of FreeNoise is not only lower than the FVD of FIFO-Diffusion (also mentioned by the authors), but is also more stable, while there is a growing trend in the FVD of FIFO-Diffusion with time. This shows that the quality of the generated sequences degrades over time. The authors claim that this is due to the lack of motion in the videos generated with FreeNoise (line 454). Some additional metrics measuring both the temporal consistency and the motion magnitude would better illustrate the tradeoff.\n\n[1] Zhang et al., Tedi: Temporally-entangled diffusion for long-term motion synthesis, arxiv 2023.\n\n[2] Ruhe et al., Rolling Diffusion Models, arxiv 2024.\n\nQUESTIONS:\n1) **Latent partitioning**: The motivation to do latent partitioning is to decrease the training/inference gap by making the timestamp differences within the window smaller. However, by increasing the size of the window by n the number of frames a certain latent interacts with throughout the denoising process also increases by the factor of n. This also could positively influence the temporal consistency of the generated videos. Therefore, it is not completely clear, whether the improved quality comes from the decreased training/inference gap or from the increased window size. Could the authors provide more details regarding this?\n\n2) **Lookahead denoising**: It is not clear from the paper, how exactly the ground truth score is obtained in eq. 7. Could the authors explain this better?\n\nIn the current form I am leaning to rejecting the paper, but I would love to raise my score, if the authors address my concerns (in Weaknesses and in Questions) in the rebuttal.\n\nLIMITATIONS:\nMore clarification would be preferred in the limitations section (see Weaknesses). Are there other limitations? E.g. violation of some basic assumptions made in the paper?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Paper Overview:**\n\nThis paper introduces the FIFO-diffusion pipeline designed for training-free autoregressive video generation. \n- The core method, ***diagonal denoising***, processes latents at varying denoising levels sequentially. \n- To enhance the video generation capabilities with more extensive denoising steps, the authors have implemented ***latent partitioning***, thereby increasing the denoise window size. \n- This modification, however, presents a challenge in maintaining consistency across video partitions, which the authors address through the approach of ***lookahead denoising***, albeit at the cost of doubling the computation.\n\n\n**Personal Reflection:**\n\nInitially, I was very intrigued by this paper. However, upon a detailed review, I noticed some areas that could benefit from improvement. While the foundational ideas are quite promising, it's important to note that the paper does not adequately cite previous works related to its core techniques.\n\nSTRENGTHS:\n1. The paper is articulate and well-structured, making it accessible and informative.\n2. Visual aids like charts and figures effectively illustrate the achieved results and methodologies.\n\nWEAKNESSES:\n**Methods:**\n1. The diagonal denoising technique bears a close resemblance to the previously work [Rolling diffusion model](https://arxiv.org/abs/2402.09470) (released at 12th Feb 2024), which unfortunately is not cited, raising concerns about the novelty.\n2. The method section heavily focuses on latent partitioning and lookahead denoising. But I think there should be some simpler solutions to decoupling the denoising steps from the window size.\n3. The concept of lookahead denoising seems similar to strategies employed in [MultiDiffusion](https://arxiv.org/abs/2302.08113) techniques, specifically in managing noise overlap to maintain partition consistency. However, there is no citation of this work, which might be seen as an oversight in acknowledging related prior work.\n\n**Result:**\n1. The core mechanism, \"diagonal denoising,\" lacks sufficient experimental explanation, especially regarding its efficacy in training-free contexts.\n2. Table 1 compares efficiency and computational costs across different FIFO settings but fails to include video quality metrics. This omission makes it unclear which setting offers the best balance of performance and quality, raising questions about the advantage of using more GPUs if fewer can achieve faster results.\n\nQUESTIONS:\nI am curious about how the rolling method initiates in the absence of historical video data. Could you explain how the process starts when there is no prior video latent available?\n\nLIMITATIONS:\nThe authors have discussed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven a “base” diffusion based video generation model, this paper shows how one can use “diagonal denoising” to generate arbitrarily long videos which have similar visual quality as the base model and obey conditioning.  Notably, the proposed method does not require any training which makes it applicable in principle to any video diffusion model.  The idea is to input a sequence of frames with a sequence of different noise scales — the denoiser is then trained to “map” these frames at noise levels [tau_1… tau_f] -> [tau_0 … tau_{f-1}].  Frames are kept in a queue — they are enqueued at noise scale tau_f and and by applying the denoiser iteratively on this queue they eventually reach noise level tau_0=0 at which point they are dequeued (and “done”).  Some tricks are proposed for making this basic idea more accurate (such as partitioning the noise scale range and applying the above trick separately for the different partitions.\n\nSTRENGTHS:\nThis method is a proof-of-concept that given a base video diffusion model capable of generating a short clip, we can always use that same model to generate arbitrarily long clips without needing to retrain.\n\nIt is not the first such paper (two competitors include Freenoise and Gen-L) but this paper is particularly simple compared to the other methods and outperforms Freenoise in user-studies (which outperforms Gen-L).  I’m admittedly less familiar with Freenoise and Gen-L, but I suspect that compared to these other papers, this approach may also be more architecture-independent as it seems applicable to any video diffusion model; moreover, compared to Freenoise, this approach does not require attending to all frames, so it runs in bounded memory.\n\nWEAKNESSES:\nNot a weakness per-se but a recommendation:\nOne thing that would make this paper stronger would be to include long-video FVD evaluations on datasets like UCF and Kinetics (see, e.g. the recent TECO paper).  This would establish a baseline of performance on those datasets showing a “theoretical maximum FVD” that can be achieved by using a model that was not trained explicitly on long videos and would be a valuable contribution to the community.\n\nAnother suggestion for readibility is to improve the description of the method in the introduction.  It is hard to understand why there would be a train-test discrepancy until Section 4 and I kept on thinking that the authors were over-claiming that it wasn’t necessary to train until I understood the details of the approach.\n\nQUESTIONS:\nThere is a recent paper on “Rolling diffusion” by Ruhe et al that seems related to this work.  I wonder if authors could comment on the relevance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}