IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis

Authors: Shitong Shao, zikai zhou, Bai LiChen, Haoyi Xiong, Zeke Xie
Primary area: generative models
Keywords: IV-mixed Sampler, Video Synthesis, Inference-heavy Algorithms, Training-free

Abstract:
Exploring suitable solutions to improve performance by increasing the computational cost of inference in visual diffusion models is a highly promising direction. Sufficient prior studies have demonstrated that correctly scaling up computation in the sampling process can successfully lead to improved generation quality, enhanced image editing, and compositional generalization. While there have been rapid advancements in developing inference-heavy algorithms for improved image generation, relatively little work has explored inference scaling laws in video diffusion models (VDMs). Furthermore, existing research shows only minimal performance gains that are perceptible to the naked eye. To address this, we design a novel training-free algorithm IV-Mixed Sampler that leverages the strengths of image diffusion models (IDMs) to assist VDMs surpass their current capabilities. The core of IV-Mixed Sampler is to use IDMs to significantly enhance the quality of each video frame and VDMs ensure the temporal coherence of the video during the sampling process. Our experiments have demonstrated that IV-Mixed Sampler achieves state-of-the-art performance on 4 benchmarks including UCF-101-FVD, MSR-VTT-FVD, Chronomagic-Bench-150/1649, and VBench. For example, the open-source Animatediff with IV-Mixed Sampler reduces the UMT-FVD score from 275.2 to 228.6, closing to 223.1 from the closed-source Pika-2.0.

Decision: accept (poster)
Reviewer scores: 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730577425148

thread_type:
official_review

<thread_object>
creator: reviewer_ypzs

<content>
summary:
The paper introduces IV-Mixed Sampler, a novel algorithm designed to enhance video synthesis by leveraging the strengths of both Image Diffusion Models (IDMs) and Video Diffusion Models (VDMs). The core innovation is the use of IDMs to improve the quality of individual video frames while maintaining temporal coherence through VDMs during the sampling process. The authors claim state-of-the-art performance on four benchmarks, including UCF-101-FVD, MSR-VTT-FVD, Chronomagic-Bench-150, and Chronomagic-Bench-1649. The paper also provides a theoretical analysis of the IV-Mixed Sampler and its transformation into a standard inverse ODE process, as well as an exploration of the design space for hyperparameters.

strengths:
Originality: The concept of combining IDMs and VDMs to enhance video synthesis is innovative. The paper addresses a significant gap in the field by improving the visual quality of synthesized videos while preserving temporal coherence.

Quality: The paper is well-structured, with a clear problem statement, methodology, experimental validation, and conclusion. The theoretical analysis provides a solid foundation for the proposed algorithm.

Clarity: The authors have done an excellent job of explaining complex concepts in a clear and concise manner. The figures and tables are well-designed and aid in understanding the content.

weaknesses:
Evaluation Metrics: The paper employs benchmarks like UCF-FVD and MSR-VTT-FVD, which can not accurately assess text-to-video generation models. More suitable benchmarks like EvalCrafter or Vbench are needed for a comprehensive evaluation.

Demonstration Insufficiency: The provided demos do not clearly demonstrate the proposed method's superiority over baselines, requiring more compelling examples to showcase improvements.

Illustration Clarity: Figure 3's illustrations are not clear. The images presented do not effectively convey the differences in quality and temporal coherence, making it difficult for readers to grasp the paper's points regarding the performance of various samplers. High-quality, clear visualizations are crucial for helping readers understand the nuances of video synthesis methods, and the paper would benefit from improved clarity in its visual aids.

Limited Applicability: The method's compatibility with state-of-the-art video generation models, which often use distinct VAEs from image models, is limited. This restricts its potential applications and prospects in the field.

questions:
1. What are the detailed reasons that we can't use different VAEs for IDM and VDM when adapting the proposed method? Do you have experiment results about this?
2. How does the IV-Mixed Sampler perform as the length and complexity of the video sequences increase? Are there any scalability issues that the authors have identified?
3. Potential Limitations: Are there any specific scenarios or types of video content where the IV-Mixed Sampler might underperform? If so, how might these limitations be addressed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618241940

thread_type:
official_review

<thread_object>
creator: reviewer_zana

<content>
summary:
The paper proposes IV-Mixed Sampler, which is to combine image diffusion models (IDMs) with video diffusion models (VDMs) to get the benefit of higher image quality of IDM and good temporal consistency of VDM. This is achieved by sampling from x_t to x_{t+∆t} and from x_{t+∆t} to x_t using IDM and VDM

strengths:
+ The paper combines theoretical proof with practical implementations, and is relatively well-rounded
+ The proposed method seems to make intuitive sense

weaknesses:
The writing of the paper can be improved. Here are several comments but not just limited to these points. Overall the presentation is a bit confusing.
- Why do the paper keep mentioning OpenAI's strawberry in Abstract and Intro, which is closed-sourced and no paper or technical detais was released? How is this important or even related to motivating the paper?
- Figure 2 is confusing and should not be put in the intro. This is more like ablation studies, and it is made rather confusing, especially authors also start to talk about "R-" in the intro out of nowhere (line 94), and the starts to discussion "I-" and "V-" in line 107 without any explanation
- The use of "go", "back", "begin", "end" etc in the equations are also confusing
- Plots like Fig 6 are also hard to read

The results of the model is decent but not very strong. For example, FreeInit is significantly better than ours for UMT-FVD on ModelScope-T2V

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730822218937

thread_type:
official_review

<thread_object>
creator: reviewer_vigw

<content>
summary:
The paper presents a novel sampling scheme for video diffusion models that leverages pre-train image diffusion models, as they generally have higher visual fidelity compared to current open source video models. The sampling method takes steps forward / backward (DDIM inversion / DDIM) in the diffusion model, alternating between the score functions of the image and video models. Results show high fidelity visual quality while retaining consistent motion in generated video.

strengths:
* The paper is generally clear, and well written
* The proposed method is novel and interesting
* Results seem to show good improvement in generation quality of the videos, while retaining consistent motion
* Main experiments and ablations are thorough and show the benefits and tradeoffs of different instantiations of the proposed method

weaknesses:
* The sampling process requires both I/V models to be in the same underlying latent space, which may be restrictive. How would this method also be used in cases where there is temporal downsampling in the video latent space (as this this is a very common video generation architecture)?
* It is unclear how useful / relevant this method may be ~6+ months from now, as the main motivation of the paper is leveraging the lack of good open source video models, and public video datasets will get better (e.g. OpenVid10M [1]), and better video generation models will be released (e.g. Mochi-1 [2] as of recently).

[1] https://huggingface.co/datasets/nkp37/OpenVid-1M
[2] https://huggingface.co/genmo/mochi-1-preview

questions:
* What is the variance of the quantitative metrics (e.g. FVD in Tables 1 and 2)? The values are pretty close and it’s unclear  if the results are statistically significant
* It would be nice to have more video samples to look at (e.g. on an anonymous website). There is only one set of videos in the supplementary, and it is hard to see motion for ones in the paper appendix.
* Does this method still work in scenarios using distilled models? E.g. 1 step or 2 step generation
</content>
</thread_object>

</thread>
<thread>
creation date:
1732120520611

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate all reviewers for their positive comments:

1. The paper is generally clear, and well written. This paper is also well-structured, with a clear problem statement, methodology, experimental validation, and conclusion **[ViGW, ypZs]**.

2. The method  is novel and interesting, which seems to make intuitive sense **[ViGW, ZanA]**.

3. The paper combines theoretical proof with practical implementations, and is relatively well-rounded **[ZanA]**.

4. Main experiments and ablations are thorough and show the benefits and tradeoffs of different instantiations of the proposed method **[ViGW]**.

We also make the revision following your constructive suggestions:

1.  Exploring the potential of IV-mixed Sampler on more advanced VDMs (e.g., Mochi-1) and addressing scalability issues for longer videos **[ViGW, ypZs]**.

2. Resolving presentation issues throughout the paper, including figures, explanations, and definitions of formulas **[ZanA]**, evaluating the IV-mixed Sampler on VBench **[ypZs]**.

3. Assessing the performance of the IV-mixed Sampler on the distilled model **[ViGW]**. 

4. Further validation on VBench **[ypZs]**.

5. Adding an anonymous page to present a visualization of IV-mixed Sampler **[ViGW]**. 

6. Add the standard deviation to indicate IV-mix sampler is significant **[ViGW]**. 

7. Replace IDM to allow IV-mixed Sampler to beat FreeInit on Modelscope-T2V **[ZanA]**.

These suggestions will certainly help us improve the quality of the paper, and we will incorporate all of them in our revision. Below, we summarize our responses and provide further clarifications to the questions from each reviewer.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes IV-mixed Sampler, a method for improving the quality and capabilities of video diffusion models by integrating an image diffusion model in the sampling process (since image models are known to have higher visual quality)---effectively mixing the spatial/visual fidelity of an image model with the temporal coherence of a video model. Strengths include the practical value of improving video synthesis without requiring training and comprehensive experimental validation showing improvements over existing methods (state-of-the-art performance across multiple benchmarks). Reviewers questioned whether the improvements were significant enough compared to existing approaches, and raised concerns about limited applicability (since the method requires two models that have been trained on the same latent space). Additional weaknesses include insufficient real-world testing and unclear explanations of some technical components.

Reviews were mixed, but tend positive. Given the reasonable method and compelling results, I am inclined to recommend acceptance.