I4VGen: Image as Free Stepping Stone for Text-to-Video Generation

Authors: Xiefan Guo, Jinlin Liu, Miaomiao Cui, Liefeng Bo, Di Huang
Primary area: generative models
Keywords: Text-to-Video, Video Diffusion Models, Video Synthesis

Abstract:
Text-to-video generation has trailed behind text-to-image generation in terms of quality and diversity, primarily due to the inherent complexities of spatio-temporal modeling and the limited availability of video-text datasets. Recent text-to-video diffusion models employ the image as an intermediate step, significantly enhancing overall performance but incurring high training costs. In this paper, we present I4VGen, a novel video diffusion inference pipeline to leverage advanced image techniques to enhance pre-trained text-to-video diffusion models, which requires no additional training. Instead of the vanilla text-to-video inference pipeline, I4VGen consists of two stages: anchor image synthesis and anchor image-augmented text-to-video synthesis. Correspondingly, a simple yet effective generation-selection strategy is employed to achieve visually-realistic and semantically-faithful anchor image, and an innovative noise-invariant video score distillation sampling (NI-VSDS) is developed to animate the image to a dynamic video by distilling motion knowledge from video diffusion models, followed by a video regeneration process to refine the video. Extensive experiments show that the proposed method produces videos with higher visual realism and textual fidelity. Furthermore, I4VGen also supports being seamlessly integrated into existing image-to-video diffusion models, thereby improving overall video quality.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730272435909

thread_type:
official_review

<thread_object>
creator: reviewer_tid5

<content>
summary:
The paper proposes a novel method to initialize the noise for T2V. To endow the noise with motion prior, the paper take the idea of SDS in T-to-3D in a video generation way, which is considered to be novel and inspiring to the community.

strengths:
The paper focuses on the video generation quality from a noise initialization viewpoint, which is hot and also vital to AI-generated content. The authors try to use the off-the-shelf T2I and T2V models in a novel way, to be specific, anchor image synthesis using T2I and motion prior using T2V in a novel SDS way.

weaknesses:
Generally speaking, video generation is notoriously for its lengthy inference cost. Noise initialization adds more cost, which is not practical. The authors are thus encouraged to add more discussion. Besides, the experiments can be more consolidated to enhance the authors' claim. Please refer to the question part for details.

questions:
I have several concerns as follow
1. More proof of the anchor image synthesis. Generally speaking, anchor image act as a low frequency component to mitigate the information leak. A question then rise that whether the anchor image synthesis is necessary. In ablation study, the authors use an example in Fig.5 to validate the assumption. Meanwhile, the authors also confirms that without the generation-selection strategy, the proposed method still performs well. Can the authors give more examples to support this claim?
2. In User study, the 20 volunteers with expertise in image and video processing participated. Does the expert bias exist? that is to say, the participants are more tolerant to the defects of the methods? Will the ordinary participants from a consumer's perspective be better?
3. It's a trend that video generation tends to produce more diverse content with long duration, for example, the large camera pose change. Will the proposed method still work?
4. The proposed method further increase the inference cost. More comprehensive measures are better presented from a practical perspective.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538046552

thread_type:
official_review

<thread_object>
creator: reviewer_hdjr

<content>
summary:
This paper presents I4VGEN, a novel video diffusion inference pipeline that enhances pre-trained text-to-video models without additional training. It tackles the complexities of spatio-temporal modeling by utilizing advanced image techniques in two stages: first, synthesizing anchor images with a strategy to ensure visual realism and semantic accuracy; second, augmenting these images to generate videos through a noise-invariant video score distillation sampling (NI-VSDS) method. This process also includes a video regeneration step for refinement. Experiments show that I4VGEN significantly improves the visual quality and textual accuracy of generated videos and can be easily integrated into existing models, boosting overall video quality.

strengths:
1. This paper introduces a training-free pipeline called I4VGen to improve the performance of text-to-video diffusion models throught image reference information.
2. A simple yet effective generation-selection strategy is proposed to obtain high-quality-images, while a noise-invariant video score distillation sampling is introduced for image animation.
3. Extensive experiments show that the proposed method comsiderably outperforms the performance of video diffusion baselines in terms of video quliaty.

weaknesses:
1. The technical contributions of the paper are somewhat limited. The proposed noise-invariant video score distillation only modifies some hyper-parameters of the original SDS techinque.
2. Compared to the baseline results, the video actions enhanced using the proposed method in this paper are minimal or essentially stationary. The metrics in Table 1 also show that the proposed method heavily harm the dynamic degree of generated videos.
3. AnimateDiff relies on high-quality LoRAs to improve the quality and consistency of generated videos. Please provide generated videos of AnimateDiff with high-quality LoRAs for a fair comparison.

questions:
1. Concerns about inference time. The proposed method consists of two stages: anchor image synthesis and anchor image-augmented video synthesis. The reviewer wants to know whether the time in the table includes the time used in the first stage.
2. Low-quality face video in Fig.6. In 1st row of Fig6, SparseCtrl produces a video in extremely low quality, could the authors explain reasons?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556500036

thread_type:
official_review

<thread_object>
creator: reviewer_dmyx

<content>
summary:
This paper proposes a video diffusion inference pipeline that leverages image generation techniques to enhance a pre-trained text-to-video (T2V) diffusion model. Instead of directly generating videos from noise, the method first utilizes a text-to-image (T2I) model to generate a high-quality anchor image. This image is then used to produce an initial video via Score Distillation Sampling (SDS) through the T2V model. A regeneration process is adopted to refine it, resulting in the final video. Experiments have been conducted to evaluate the effectiveness of the proposed method both qualitatively and quantitatively.

strengths:
1. The proposed method uses a pre-trained image generation model to improve frame quality in text-to-video generation, which is helpful for high-quality video generation.
2. The presented results demonstrate good quality.

weaknesses:
1. The proposed method appears to integrate the T2I model with SDS distillation for video generation, and the contribution seems incremental.
2. The motion observed in Fig. 1 appears to be smaller compared to the baselines.
3. There is a lack of analysis for different regeneration steps.

questions:
1. What are the effects of using different image generation models? Can better video quality be achieved if a better image model (e.g., FLUX) is used to generate anchor images?
2. Will the SDS degrade image quality? It would be better to provide quantitative results in Table 2 to show its effects.
3. In Fig. 8, the prompt for the last video is mistaken.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691546381

thread_type:
official_review

<thread_object>
creator: reviewer_vp1p

<content>
summary:
This paper presented a two-stage text-to-Video Generation method I4VGEN, i.e., anchor image synthesis and anchor image-augmented text-to-video synthesis. Experiments are provided to assess its effectiveness in text2video generation and enhancing I2V methods.

strengths:
+ Two-stage text-to-Video Generation method I4VGEN.
+ Can be integrated into existing image-to-video diffusion models.

weaknesses:
- The integration with existing image-to-video diffusion models is interesting, but the authors are suggested to combined with more I2V models, especially several recent ones.
- More ablation studies are required to show whether the anchor image selection and the NI-VSDS  are optimal.
- In the bottom of Fig. 6, albeit better image quality, it seems that the motion of the proposed method is smaller than that by SparseCtrl. More experiments are suggested to assess this aspect.

questions:
- The integration with existing image-to-video diffusion models is interesting, but the authors are suggested to combined with more I2V models, especially several recent ones.
- More ablation studies are required to show whether the anchor image selection and the NI-VSDS  are optimal.
- In the bottom of Fig. 6, albeit better image quality, it seems that the motion of the proposed method is smaller than that by SparseCtrl. More experiments are suggested to assess this aspect.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733281112815

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Thank you to AC for supporting the discussion, and thanks to the reviewers for their time and thoughtful comments and feedback!

Best regards, 

The authors of Paper #7261
</content>
</thread_object>

</thread>


Metareview:
metareview:
While the proposed I4VGen method offers practical improvements for text-to-video generation, concerns about its novelty (dmyx), incremental technical contributions (hdjr) and practicality (tiD5) remain. Despite efforts in the rebuttal to address these issues, some reviewers feel that the contributions, particularly NI-VSDS, are minor modifications of SDS. In addition, the trade-off between dynamic motion and temporal consistency raises questions about the method's general applicability. Given the borderline scores and unresolved concerns, the decision is to reject. The authors are encouraged to refine the contributions and address the reviewers’ feedback for future submissions.