I4VGen: Image as Free Stepping Stone for Text-to-Video Generation

## Abstract

Abstract Text-to-video generation has trailed behind text-to-image generation in terms of quality and diversity, primarily due to the inherent complexities of spatio-temporal modeling and the limited availability of video-text datasets. Recent text-to-video diffusion models employ the image as an intermediate step, significantly enhancing overall performance but incurring high training costs. In this paper, we present I4VGen , a novel video diffusion inference pipeline to leverage advanced image techniques to enhance pre-trained text-to-video diffusion models, which requires no additional training. Instead of the vanilla text-to-video inference pipeline, I4VGen consists of two stages: anchor image synthesis and anchor image-augmented text-to-video synthesis. Correspondingly, a simple yet effective generation-selection strategy is employed to achieve visually-realistic and semantically-faithful anchor image, and an innovative noise-invariant video score distillation sampling (NI-VSDS) is developed to animate the image to a dynamic video by distilling motion knowledge from video diffusion models, followed by a video regeneration process to refine the video. Extensive experiments show that the proposed method produces videos with higher visual realism and textual fidelity. Furthermore, I4VGen also supports being seamlessly integrated into existing image-to-video diffusion models, thereby improving overall video quality.

## 1 Introduction

Recent advances in large-scale text-to-image diffusion models (Esser et al., ; Balaji et al., ; Ramesh et al., ; Nichol et al., ; Saharia et al., ; Feng et al., ; Gu et al., ; Xue et al., ) have demonstrated the capability to generate diverse and high-quality images from extensive web-scale image-text pair datasets. Efforts to extend these diffusion models to text-to-video synthesis (Ho et al., ; Zhou et al., ; Chen et al., ; Singer et al., ; Wang et al., ; ; Blattmann et al., ; Girdhar et al., ; Guo et al., ; Bao et al., ) have involved leveraging video-text pairs and temporal modeling. However, text-to-video generation remains inferior to image counterpart in terms of both quality and diversity, primarily due to the complex nature of spatio-temporal modeling and the limited size of video-text datasets, which are often an order of magnitude smaller than image-text datasets.

This paper explores a novel video diffusion inference pipeline that leverages advanced image techniques to enhance pre-trained text-to-video diffusion models, focusing on the following two insights:

Image conditioning for text-to-video generation. Recent methods (Blattmann et al., ; Zhang et al., ; Girdhar et al., ; Chen et al., ; Li et al., ; Hu et al., ) have adopted image-guided text-to-video generation, where an initial image generation step significantly enhances video output quality. This paradigm benefits from the strong capabilities of text-to-image models by using the generated images as detailed references for video synthesis. While effective, these approaches incurs additional high training costs. This paper builds on this insight but innovates by designing a novel video diffusion inference pipeline to leverage image information, thereby enhancing text-to-video generation performance without additional training expense.

Figure: Figure 2: Illustration of non-zero terminal signal-to-noise ratio. We employ t-SNE to visualize the distributions of pure Gaussian noise, real video, and noisy video at the timestep $T$, where each data point represents an independently sampled noise point or video frame. The noise schedule of AnimateDiff (Guo et al., ) is used, and all operations are performed in the latent space of the video autoencoder. (a) The distribution of pure Gaussian noise exhibits a disordered and diffuse nature; (b) real videos are temporally-correlated and different videos can be clearly distinguished from each other; (c) noisy videos preserve a certain degree of temporal correlation and maintain separability between different videos; (d) sampled videos for visualization.
Refer to caption: x2.png

Zero terminal-SNR noise schedule. A prevalent issue in diffusion models is the non-zero terminal signal-to-noise ratio (SNR) (; Lin et al., ). The mismatch between the training phase, where residual signals persist in noisy videos at the terminal diffusion timestep $T$, and the inference phase, which uses pure Gaussian noise at the timestep $T$, creates a gap that degrade the model performance. As illustrated in Fig. 2, noisy videos exhibit temporal correlation that is distinctly different from the independent and identically distributed pure Gaussian noise. This paper is dedicated to reconfiguring the inference pipeline to circumvent this issue.

Motivated by these insights, we propose a novel video diffusion inference pipeline, called I4VGen, which enhances pre-trained text-to-video diffusion models by incorporating image information into the inference process. This method requires no additional learnable parameters and training costs, and can be seamlessly integrated into existing text-to-video diffusion models, circumventing the non-zero terminal SNR issue and improving output quality.

Specifically, instead of the vanilla text-to-video inference pipeline, which fails to leverage image reference information, I4VGen decomposes the inference process into two stages: anchor image synthesis and anchor image-augmented text-to-video synthesis. For the former, a simple yet effective generation-selection strategy is introduced, which involves synthesizing candidate images and selecting the most suitable one using a reward-based mechanism, thereby obtaining a visually-realistic anchor image that is closely aligned with the text prompt. For the latter, we develop an innovative noise-invariant video score distillation sampling (NI-VSDS) to animate the anchor image to a dynamic video by extracting motion knowledge from text-to-video diffusion models, followed by a video regeneration process, *i.e.*, diffusion-denoising, to refine the video. This inference pipeline avoids the issue of non-zero terminal SNR.

Extensive quantitative and qualitative analyses demonstrate that I4VGen can be effectively applied to various text-to-video diffusion models, significantly improving the temporal consistency, visual realism, and semantic fidelity of the synthesized videos (see Fig. 1). Moreover, our method can also be seamlessly integrated into existing image-to-video diffusion models, thereby enhancing the temporal consistency and visual quality of the generated videos (see Fig. 6).

The main novelties and contributions are as follows:

- •
We propose a novel video diffusion inference pipeline, called I4VGen, which enhances pre-trained text-to-video diffusion models by incorporating image reference information into the inference process, without requiring additional training or learnable parameters.
- •
We employ a simple yet effective generation-selection strategy to achieve high-quality image, and design a novel noise-invariant video score distillation sampling for image animation.
- •
We comprehensively evaluate our approach with representative
text-to-video diffusion models, and demonstrate I4VGen significantly improves the quality of generated videos. Furthermore, I4VGen can also be adapted to image-to-video diffusion models, leading to improved results.

## 2 Preliminaries

Video diffusion models. Aligned with the framework of image diffusion models, Video diffusion models (VDMs) predominantly utilize the paradigm of latent diffusion models (LDMs). Unlike traditional methods that operate directly in the pixel space, VDMs function within the latent space defined by a video autoencoder. Specifically, a video encoder $\mathcal{E}\left(\cdot\right)$ learns the mapping from an input video $\mathbf{v}\in\mathcal{V},{\mathbf{v}}=\{{\mathbf{f}}^{1},{\mathbf{f}}^{2},
\cdots,{\mathbf{f}}^{F}\}$ to a latent code $\mathbf{z}=\mathcal{E}\left(\mathbf{v}\right)=\{{\mathbf{z}}^{1},{\mathbf{z}}^
{2},\cdots,{\mathbf{z}}^{f}\}$. Subsequently, a video decoder $\mathcal{D}(\cdot)$ reconstructs the input video, aiming for $\mathcal{D}\left(\mathcal{E}\left(\mathbf{v}\right)\right)\approx\mathbf{v}$. Typically, image autoencoder is used in a frame-by-frame processing manner instead of the video one, where $F=f$.

Upon training the autoencoder, a Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., ) is employed within the latent space to generate a denoised version of an input latent $\mathbf{z}_{t}$ at each timestep $t$. During denoising, the diffusion model can be conditioned on additional inputs, such as a text embedding $\mathbf{c}=f_{\text{CLIP}}\left(\mathbf{y}\right)$ generated by a pre-trained CLIP text encoder (Radford et al., ), corresponding to the input text prompt $\mathbf{y}$. The DDPM model $\epsilon_{\theta}(\cdot)$, a 3D U-Net parametrized by $\theta$, optimizes the following loss:

$$ $\mathcal{L}=\mathbb{E}_{\mathbf{z}\sim\mathcal{E}\left(\mathbf{v}\right), \mathbf{c}=f_{\text{CLIP}}\left(\mathbf{y}\right),\epsilon\sim\mathcal{N}\left (\mathbf{0},\mathbf{1}\right),t}\left[\|\epsilon-\epsilon_{\theta}\left( \mathbf{z}_{t},\mathbf{c},t\right)\|_{2}^{2}\right],$ (1) $$

During inference, a latent variable $\mathbf{z}_{T}$ is sampled from the standard Gaussian distribution $\mathcal{N}(\mathbf{0},\mathbf{1})$ and subjected to sequential denoising procedures of the DDPM to derive a refined latent $\mathbf{z}_{0}$. This denoised latent $\mathbf{z}_{0}$ is then fed into the decoder to synthesize the corresponding video $\mathcal{D}\left(\mathbf{z}_{0}\right)$.

Figure: Figure 3: Illustration of I4VGen. I4VGen is a novel video diffusion inference pipeline, which enhances pre-trained text-to-video diffusion models by incorporating image reference information into the inference process. Instead of the vanilla text-to-video inference pipeline, I4VGen consists of two stages: (1) anchor image synthesis and (2) anchor image-augmented text-to-video synthesis. Firstly, a simple yet effective generation-selection strategy is applied to synthesize candidate images and select the most suitable image using a reward-based mechanism, thereby obtaining high-quality anchor image. Subsequently, an innovative noise-invariant video scoring distillation sampling (NI-VSDS) is developed, which extracts motion prior from the text-to-video diffusion model to animate the anchor image into dynamic video, followed by a video regeneration process to refine the video.
Refer to caption: x3.png

Score distillation sampling. Score distillation sampling (SDS) (Poole et al., ; Wang et al., ) employs the priors of pre-trained text-to-image models to facilitate text-conditioned 3D generation. Specifically, given a pre-trained diffusion model $\epsilon_{\theta}(\cdot)$ and the conditioning embedding $\mathbf{c}=f_{\text{CLIP}}\left(\mathbf{y}\right)$ corresponding to the text prompt $\mathbf{y}$, SDS optimizes a set of parameters $\phi$ of a differentiable parametric image generator $\mathcal{G}(\cdot)$ (*e.g.*, NeRF (Mildenhall et al., )) using the gradient of the SDS loss $\mathcal{L}_{\text{SDS}}$:

$$ $\nabla_{\phi}\mathcal{L}_{\text{SDS}}=w(t)\left(\epsilon_{\theta}(\mathbf{z}_{ t},\mathbf{c},t)-\epsilon\right)\frac{\partial\mathbf{x}}{\partial\phi},$ (2) $$

where $\epsilon$ is sampled from $\mathcal{N}\left(\mathbf{0},\mathbf{1}\right)$, $\mathbf{x}$ is an image rendered by $\mathcal{G}$, $\mathbf{z}_{t}$ is obtained by adding Gaussian noise $\epsilon$ to $\mathbf{x}$ corresponding to the timestep $t$ of the diffusion process, $w(t)$ is a constant that depends on the noising schedule. Inspired by this method, we proposes a noise-invariant video score distillation sampling (NI-VSDS) strategy to efficiently harness the motion prior learned by the text-to-video diffusion model.

## 3 I4VGen

This section introduces I4VGen, a novel video diffusion inference pipeline designed for enhancing the capabilities of pre-trained text-to-video diffusion models. As illustrated in Fig. 3, we factorize the inference process into two stages: (1) anchor image synthesis to generate the anchor image $\mathbf{x}$ given the text prompt $\mathbf{y}$, and (2) anchor image-augmented video synthesis to generate the video $\mathbf{v}$ by leveraging the text prompt $\mathbf{y}$ and the anchor image $\mathbf{x}$. This section provides the detailed explanations of both stages in Sec. 3.1 and 3.2, respectively.

### 3.1 Anchor image synthesis

The goal of this stage is to synthesize visually-realistic anchor images $\mathbf{x}$ that accurately correspond to the given text prompts $\mathbf{y}$. This image serves as a foundation to provide appearance information for enhancing the performance of the subsequent video generation. As illustrated in Fig. 3 (Top), a simple yet effective generation-selection pipeline is employed to produce the anchor image, which consist of candidate images synthesis and reward-based anchor image selection.

Candidate images synthesis. Instead of generating a single image, our approach produces a set of candidate images to ensure the selection of the best example. Utilizing a pre-trained image diffusion model $\mathcal{D}_{\text{img}}(\cdot)$, we construct the candidate image set as follows:

$$ $\displaystyle{\mathbf{x}}_{1},{\mathbf{x}}_{2},\cdots,{\mathbf{x}}_{N}= \mathcal{D}_{\text{img}}\left(\mathbf{y},\mathbf{z}_{1}\right),\mathcal{D}_{ \text{img}}\left(\mathbf{y},\mathbf{z}_{2}\right),\cdots,\mathcal{D}_{\text{ img}}\left(\mathbf{y},\mathbf{z}_{N}\right),$ (3) $$

where $N$ denotes the number of candidate images, and $\mathbf{z}_{i}$ represents Gaussian noise.

Reward-based anchor image selection. With the help of the image reward model $\mathcal{R}(\cdot)$ (Xu et al., ), a promising automatic text-to-image evaluation metric aligned with human preferences, the candidate image with the highest reward score $s$ is selected as the anchor image $\mathbf{x}$, as defined by:

$$ $\mathbf{x}=\mathbf{x}_{i},\quad\text{where}\ i=\mathop{\arg\max}\limits_{i}s_{ i}=\mathop{\arg\max}\limits_{i}\mathcal{R}\left(\mathbf{x}_{i}\right).$ (4) $$

The generation-selection design facilitates the acquisition of a high-quality anchor image, particularly beneficial for complex text prompts (see Fig. 5). Notably, our method accommodates both user-provided and retrieved images, extending its applicability to a variety of custom scenarios, as discussed in Sec. 4.5.

Figure: Figure 4: Qualitative comparison. Each video is generated with the same text prompt and random seed for all methods. Our approach significantly improves the quality of the generated videos while showing excellent alignment with text prompts.
Refer to caption: x4.png

### 3.2 Anchor image-augmented video synthesis

Upon obtaining the anchor image $\mathbf{x}$, we replicate it $F$ times to create an initial static video $\hat{\mathbf{v}}\in\mathcal{V},\hat{\mathbf{v}}=\{{\mathbf{x}},{\mathbf{x}},
\cdots,{\mathbf{x}}\}$. The goal of this stage is to convert this static video into a high-quality video reflecting the text prompt $\mathbf{y}$. As illustrated in Fig. 3 (Bottom), we introduce static video animation and video regeneration.

Static video animation. A straightforward approach to animate the static video involves applying a diffusion-denoising process to transition from the static to dynamic state. However, this approach still encounters a training-inference gap, as the text-to-video diffusion model is trained on dynamic real-world videos but tested on static videos, leading to sub-optimal motion quality due to the introduction of static priors, as discussed in Sec. 4.4.

To address this limitation, we propose a novel approach leveraging the motion prior from the pre-trained text-to-video diffusion model to animate static videos. Drawing inspiration from score distillation sampling (SDS) as introduced in (Poole et al., ; Wang et al., ), we develop the noise-invariant video score distillation sampling (NI-VSDS). Unlike vanilla SDS, which optimizes a parametric image generator, our approach directly parameterizes the static video $\hat{\mathbf{v}}$ and applies targeted optimization to it. The NI-VSDS loss function is defined as follows:

$$ $\nabla_{\hat{\mathbf{v}}}\mathcal{L}_{\text{NI-VSDS}}=w(t)\left(\epsilon_{ \theta}(\hat{\mathbf{v}}_{t},\mathbf{c},t)-\epsilon\right),$ (5) $$

where $\hat{\mathbf{v}}_{t}$ represents the noisy video at timestep $t$ perturbed by Gaussian noise $\epsilon$. Furthermore, we incorporate three strategic modifications:

- •
Instead of resampling the Gaussian noise at each iteration as in traditional SDS, we maintain a constant noise across the optimization, enhancing convergence speed.
- •
Optimization is confined to the initial stages of the denoising process, where noise levels are higher, focusing on dynamic information distillation.
- •
We implement a coarse-to-fine optimization strategy, evolving from high to low noise levels, specifically from timestep $T$ to $\tau_{\text{NI-VSDS}}$, where $T>\tau_{\text{NI-VSDS}}>0$. This approach stabilizes the optimization trajectory and yields superior motion quality.

Figure: Algorithm 1 NI-VSDS

The implementation of noise-invariant video score distillation sampling (NI-VSDS) algorithm is detailed in Algorithm 1, which outlines the process of converting a static video into a dynamic video using the defined NI-VSDS loss. Notably, we only perform a single update from timestep $T$ to $\tau_{\text{NI-VSDS}}$, requiring fewer than 50 iterations, this is a significant reduction compared to the thousands of iterations typically required for text-to-3D synthesis in SDS. $\alpha$ is a scalar that defines the step size of the gradient update. We empirically set $\tau_{\text{NI-VSDS}}=\mathsf{Int}(T\times p_{\text{NI-VSDS}})$.

**Table 1: VBench evaluation results per dimension. This table compares the performance of I4VGen with other counterparts across each of the 16 VBench dimensions.**
| Methods | Subj.<br>Cons. | Back.<br>Cons. | Tem.<br>Flick. | Moti.<br>Smo. | Dyna.<br>Degr. | Aest.<br>Qual. | Imag.<br>Qual. | Obje.<br>Class |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AnimateDiff | 87.11% | 95.22% | 95.99% | 93.12% | 74.89% | 56.07% | 64.29% | 83.69% |
| + FreeInit | 90.45% | 96.57% | 96.89% | 95.66% | 70.17% | 59.25% | 63.51% | 87.55% |
| + I4VGen | 95.17% | 97.73% | 98.51% | 96.45% | 57.72% | 64.68% | 66.18% | 92.59% |
| LaVie | 91.65% | 96.30% | 98.03% | 95.73% | 71.94% | 59.64% | 65.13% | 91.25% |
| + FreeInit | 92.32% | 96.35% | 98.06% | 95.83% | 71.11% | 59.41% | 63.89% | 89.13% |
| + I4VGen | 94.12% | 96.90% | 98.55% | 96.37% | 70.55% | 60.88% | 66.55% | 92.26% |
| Methods | Mult.<br>Obje. | Hum.<br>Acti. | Color | Spat.<br>Rela. | Scene | Appe.<br>Style | Tem.<br>Style | Over.<br>Cons. |
| AnimateDiff | 22.61% | 90.40% | 81.73% | 31.55% | 45.61% | 24.40% | 24.49% | 25.71% |
| + FreeInit | 26.92% | 93.00% | 86.39% | 30.71% | 44.61% | 23.98% | 25.03% | 25.61% |
| + I4VGen | 57.22% | 95.80% | 91.98% | 45.20% | 54.67% | 25.07% | 26.11% | 28.01% |
| LaVie | 24.02% | 94.80% | 83.64% | 26.27% | 52.89% | 23.67% | 24.94% | 27.25% |
| + FreeInit | 22.59% | 94.20% | 84.34% | 27.46% | 52.70% | 23.61% | 24.85% | 26.89% |
| + I4VGen | 32.77% | 96.20% | 88.59% | 33.81% | 55.64% | 24.35% | 25.62% | 27.68% |

Video regeneration. After animating the static video, we further enhance the appearance detail quality of the video through a diffusion-denoising process. This stage is not affected by the aforementioned training-inference gap, thereby achieving more refined generation results.

Notably, we can flexibly add noise up to any timestep $\tau_{\text{re}}$, calculated as $\tau_{\text{re}}=\mathsf{Int}(T\times p_{\text{re}})$, followed by the corresponding denoising process. This strategy not only preserves the fine appearance textures but also reduces the required denoising steps, thus streamlining the video synthesis process and elevating the overall quality of the resulting video.

## 4 Experiments

### 4.1 Experimental settings

Implementation details. I4VGen a novel video diffusion inference pipeline that leverages advanced image techniques to enhance pre-trained text-to-video diffusion models without requiring additional training, and can be seamlessly integrated into existing text-to-video diffusion models. To ascertain the efficacy and adaptability of I4VGen, we apply it to two well-regarded text-to-video diffusion models: AnimateDiff (Guo et al., ) and LaVie (Wang et al., ).

**Table 2: Ablation study. Orange highlights generation-selection, while yellow highlights NI-VSDS.**
| Methods | Subj.<br>Cons. | Back.<br>Cons. | Tem.<br>Flick. | Moti.<br>Smo. | Dyna.<br>Degr. | Aest.<br>Qual. | Imag.<br>Qual. | Obje.<br>Class |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AnimateDiff | 87.11% | 95.22% | 95.99% | 93.12% | 74.89% | 56.07% | 64.29% | 83.69% |
| + I4VGen (w/o gen.-sel.) | 94.89% | 97.80% | 98.28% | 96.99% | 55.91% | 62.23% | 64.18% | 90.95% |
| + I4VGen (w/o NI-VSDS) | 96.47% | 98.82% | 98.99% | 97.56% | 28.24% | 65.17% | 65.52% | 92.66% |
| + I4VGen | 95.17% | 97.73% | 98.51% | 96.45% | 57.72% | 64.68% | 66.18% | 92.59% |
| Methods | Mult.<br>Obje. | Hum.<br>Acti. | Color | Spat.<br>Rela. | Scene | Appe.<br>Style | Tem.<br>Style | Over.<br>Cons. |
| AnimateDiff | 22.61% | 90.40% | 81.73% | 31.55% | 45.61% | 24.40% | 24.49% | 25.71% |
| + I4VGen (w/o gen.-sel.) | 40.68% | 94.40% | 90.55% | 37.79% | 53.72% | 24.76% | 26.03% | 26.62% |
| + I4VGen (w/o NI-VSDS) | 62.84% | 94.80% | 91.95% | 47.57% | 55.80% | 24.88% | 25.72% | 27.91% |
| + I4VGen | 57.22% | 95.80% | 91.98% | 45.20% | 54.67% | 25.07% | 26.11% | 28.01% |

For AnimateDiff, the mm-sd-v15-v2 motion module(^1^11[https://github.com/guoyww/AnimateDiff](https://github.com/guoyww/AnimateDiff)), alongside Stable Diffusion v1.5, is utilized to synthesize 16 consecutive frames at a resolution of $512\times 512$ pixels for evaluation. For LaVie, the base-version(^2^22[https://github.com/Vchitect/LaVie](https://github.com/Vchitect/LaVie)) is employed to generate 16 consecutive frames at $320\times 512$ pixels for evaluation. All other inference details adhere to the original settings described in Guo et al. () and Wang et al. (), respectively. Notably, both AnimateDiff and LaVie possess inherent text-to-image generation capabilities when excluding the motion module. To avoid introducing additional GPU storage requirements, we leverage their corresponding image versions for text-to-image generation in I4VGen. For AnimteDiff, we empirically set $N=16$, $p_{\text{NI-VSDS}}=0.4$, $\alpha=1$, and $p_{\text{re}}=1$. For LaVie, we empirically set $N=16$, $p_{\text{NI-VSDS}}=0.4$, $\alpha=1$, and $p_{\text{re}}=0.8$. All experiments are conducted on a single NVIDIA V100 GPU (32 GB).

Benchmark. I4VGen is assessed using VBench (Huang et al., ), a comprehensive benchmark that evaluates video generation models across 16 disentangled dimensions, which is more authoritative than FVD. These dimensions provide a detailed analysis of generation quality from two overarching perspectives: video quality(^3^33Video quality includes 7 evaluation dimensions: Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, Dynamic Degree, Aesthetic Quality, and Imaging Quality. The first 5 evaluate temporal quality, and the last 2 evaluate frame-wise quality.), focusing on the perceptual quality of the generated videos, and video-condition consistency(^4^44Video-condition consistency includes 9 evaluation dimensions: Object Class, Multiple Objects, Human Action, Color, Spatial Relationship, Scene, Appearance Style, Temporal Style, Overall Consistency. The first 6 evaluate semantics, the 7 and 8-th evaluate style, and the 9-th evaluates overall consistency.), assessing how well the generated videos align with the provided conditions.

### 4.2 Qualitative comparison

Fig. 4 presents a comparative analysis of our results against state-of-the-art counterparts using identical text prompts and random seeds. I4VGen excels in enhancing both the temporal consistency and the frame-wise quality, alongside superior alignment with the text prompts. For instance, in the case of “playing guitar", AnimateDiff suffers from poor video quality, and FreeInit encounters an incomplete guitar in the middle of the video. In contrast, our method effectively addresses these issues, maintaining stable temporal consistency. Furthermore, while baseline methods struggle with accurate synthesis of all text-described components, *e.g.*, “NYC Times Square", I4VGen generates videos that are visually realistic and closely aligned with the text prompts by utilizing anchor images obtained by the generation-selection strategy.

### 4.3 Quantitative comparison

Objective evaluation. Following the protocols established by VBench, we evaluate I4VGen in terms of both video quality and video-text consistency. As detailed in Table 1, I4VGen outperforms all other approaches in temporal quality (higher background and subject consistency, less flickering, and better smoothness), frame-wise quality (higher aesthetic and imaging quality), and video-text consistency (greater semantics, style, and overall consistency). Although counterparts occasionally produce videos with more dynamic motion, they are often linked to inappropriate or excessive movements. I4VGen strikes a more effective balance between motion intensity and overall video quality, which is further verified in the user study.

**Table 3: User study.**
| Method | Video<br>Quality | Vid.-Cond.<br>Consistency | Overall<br>score |
| --- | --- | --- | --- |
| AnimateDiff | 6.00% | 10.67% | 6.33% |
| + FreeInit | 27.67% | 15.67% | 25.00% |
| + I4VGen | 66.33% | 73.67% | 68.67% |
| LaVie | 27.67% | 21.33% | 22.33% |
| + FreeInit | 22.67% | 18.33% | 19.67% |
| + I4VGen | 49.67% | 60.33% | 58.00% |

User study. We conduct a subjective user study involving 20 volunteers with expertise in image and video processing, with each participant answering 15 questions. Specifically, participants are asked to select the video with the highest quality across three dimensions: video quality, video-condition consistency, and overall score. As shown in Table 3, our approach outperforms the other methods favorably.

**Table 4: Inference time.**
| Method | Time |
| --- | --- |
| AnimateDiff | 21.73s |
| AnimateDiff + FreeInit | 113.67s |
| AnimateDiff + I4VGen | 53.78s |

Inference time. We define the time cost of a single denoising iteration for a video in a video diffusion model as $c$. For AnimateDiff (Guo et al., ), following the original inference setting, the time cost to generate a single 16-frame video is $25c$. FreeInit requires 5 rounds of diffusion-denoising to generate a single video, taking a time of $5\times 25c=125c$. The time cost for I4VGen to generate a single video is: $<25c$ (for synthesizing 16 candidate images) $+\ 0.6\times 25c$ (for NI-VSDS) $+\leq 25c$ (for video regeneration) $=\ <65c$ (total cost), making it more efficient compared to FreeInit. LaVie (Wang et al., ) shares the same conclusion.

We also provide the inference time for a single video in Table 4, evaluated on a single NVIDIA V100 GPU (32 GB), where 50 videos are randomly generated to obtain an average inference time. Our method performs better than FreeInit.

Figure: Figure 5: Intermediate results visualization. We provide visualizations of the candidate images with reward scores, the dynamic video, and the corresponding generated video.
Refer to caption: x5.png

### 4.4 Ablation study

On generation-selection strategy. We adopt a generation-selection strategy to create visually-realistic and semantically-faithful anchor images, which serve as a foundation for providing appearance information to enhance subsequent video generation performance. As shown in Table 2, highlighted in orange, compared to randomly synthesizing a single anchor image, the generation-selection strategy significantly improves the quality of the generated videos in terms of frame-wise quality and consistency with the text. Fig. 5 provides a visualization of the candidate images, where the reward-based selection strategy eliminates unsatisfactory images, leading to better results.

On NI-VSDS. Directly applying the video regeneration process to static videos introduces static priors, resulting in suboptimal motion quality. As shown in Table 2, highlighted in yellow, while direct diffusion-denoising improves the temporal consistency of the generated videos, it severely sacrifices the motion dynamics, adversely affecting the motion style. In contrast, our method achieves an effective balance between motion intensity and overall video quality.

Figure: Figure 6: Adaptation on SparseCtrl. I4VGen can be seamlessly integrated into SparseCtrl by replacing the anchor image with the provided image, leading to improved results.
Refer to caption: x6.png

On video regeneration. Fig. 5 visualizes the intermediate results, demonstrating that the video regeneration process is essential for refining appearance details.

Figure: Figure 7: Adaptation on real image.
Refer to caption: x7.png

### 4.5 More applications

Adaptation on real image. Our method adapts to user-provided images, as shown in Fig. 7, where we use real images as anchor images, resulting in high-fidelity videos that are semantically consistent with the real images. Notably, our approach differs from vanilla image-to-video generation, as the synthesized videos are not completely aligned with the provided images. NI-VSDS is designed to animate static videos and is implemented as a spatio-temporal co-optimization.

Adaptation on image-to-video diffusion models. I4VGen can be seamlessly integrated into existing image-to-video diffusion models by replacing the anchor images with the provided images, thereby enhancing the overall video quality. As shown in Fig. 6, integrating I4VGen into SparseCtrl (Guo et al., ) significantly improves the quality of the generated videos in terms of temporal consistency and appearance fidelity.

## 5 Conclusion

The paper introduces I4VGen, a novel video diffusion inference pipeline to leverage advanced image techniques to enhance pre-trained text-to-video diffusion models, which requires no additional learnable parameters and training costs. I4VGen decomposes the text-to-video inference process into anchor image synthesis and anchor image-augmented video synthesis. Correspondingly, a simple yet effective generation-selection strategy is applied to produce a high-quality anchor image, and an innovative noise-invariant video score distillation sampling (NI-VSDS) is designed to animate the image, followed by a video regeneration process to enhance the final output. I4VGen effectively alleviates non-zero terminal signal-to-noise ratio issues and demonstrates improved visual realism and textual fidelity when integrated with existing video diffusion models.

Limitation and discussion. I4VGen improves the video diffusion model but requires more inference cost. As discussed in Sec. 4.3, the inference time of I4VGen is over double the baseline. Enhancing inference efficiency remains a future goal, with distillation techniques as a potential approach. Furthermore, removing the generation-selection strategy can reduce inference costs to some extent. As shown in Table 2, our method still significantly outperforms the baseline under this setting. Additionally, although our method and FreeInit are orthogonal, integrating both by replacing video regeneration with FreeInit fails to produce notable benefits.

## Appendix A Related Work

Video Generative Models.
The domain of video generation has seen significant advancements through the use of Generative Adversarial Networks (GANs) (Vondrick et al., ; Saito et al., ; Tulyakov et al., ; Wang et al., ; Saito et al., ; Tian et al., ; Fox et al., ; Yu et al., ; Skorokhodov et al., ; Brooks et al., ; Shen et al., ; Wang et al., ), Variational Autoencoders (VAEs) (Mittal et al., ; Li et al., ; He et al., ), and Autoregressive models (ARs) (Yan et al., ; Ge et al., ; Wu et al., ; Hong et al., ; Villegas et al., ; Fu et al., ; Yoo et al., ; Yu et al., ). Despite these developments, synthesizing videos from text prompts remains challenging due to the complexities of modeling spatio-temporal dynamics. Recent innovations driven by the successes of diffusion models (Ho et al., ; Dhariwal & Nichol, ; Song et al., ), which have been applied effectively in image generation (Rombach et al., ; Nichol et al., ; Ramesh et al., ; Saharia et al., ; Gu et al., ; Balaji et al., ; Xue et al., ; Meng et al., ; Guo et al., ) and audio synthesis (Kong et al., ; Chen et al., ; Popov et al., ; Leng et al., ; Liu et al., ), and underscore the emergence of substantial headway (Ho et al., ; ; He et al., ; Singer et al., ; Blattmann et al., ; Yu et al., ; Ruan et al., ; Wu et al., ; Chen et al., ; ; Esser et al., ; Ge et al., ; Chen et al., ; Geyer et al., ; Ma et al., ; Wang et al., ; Zhang et al., ; ; Hu et al., ; Wang et al., ; Feng et al., ; Guo et al., ; Girdhar et al., ; Blattmann et al., ; Gupta et al., ; Wang et al., ; ; Luo et al., ) in research endeavors devoted to video synthesis from text input.

The foundational contributions of the Video Diffusion Model (VDM) (Ho et al., ) represents a milestone in leveraging diffusion models for video generation by adapting the 2D U-Net architecture used in image generation to a 3D U-Net capable of temporal modeling. Successive researches, such as Make-A-Video (Singer et al., ) and Imagen Video (Ho et al., ), expand video generation capabilities significantly. To enhance efficiency, subsequent models have transitioned the diffusion process from pixel to latent space (He et al., ; Zhou et al., ; Wang et al., ; Blattmann et al., ; ; Guo et al., ; Wang et al., ), paralleling advancements in latent diffusion for images (Rombach et al., ).

However, the direct generation of videos from text prompts remains intrinsically challenging. Recent approaches (Blattmann et al., ; Zhang et al., ; Girdhar et al., ; Chen et al., ; ; Li et al., ; Hu et al., ; Yu et al., ; Ren et al., ) have employed text-to-image synthesis as an intermediary step, enhancing overall performance. Despite these advancements, these methods still face the challenge of high computational training costs. In this study, we explore a novel training-free methodology aimed at bridging the existing gap in the field.

In addition, (Chen et al., ) (contemporary researches) introduces additional operations in the attention layer, *i.e.*, cross-frame self-attention control, to enhance the video model. However, this necessitates modifications to the model architecture, whereas our method does not.

Signal-Leak Bias.
Diffusion models are designed to generate high-quality visuals from noise through a sequential denoising process, which is consistent in both image and video diffusion models. During training, Gaussian noise corrupts the visual content, challenging the model to restore it to its original form. In the inference phase, the model operates on pure Gaussian noise, transforming it into a realistic visual content step-by-step.

Unfortunately, most existing diffusion models exhibit a disparity between the corrupted image during training and the pure Gaussian noise during inference. Commencing denoising from pure Gaussian noise in the inference phase deviates from the training process, potentially introducing signal-leak bias. For image diffusion models, (; Lin et al., ; Li et al., ) point out flaws in common diffusion noise schedules and sample steps, and propose to fine-tune the diffusion model to mitigate or eliminate the signal-leak bias during training, leading to improved results. (Everaert et al., ) attempts to exploit signal-leak bias to achieve more control over the generated images. For video diffusion models, this issue becomes more pronounced. (Wu et al., ; Ma et al., ) invert the retrieved video or generated low-quality to construct initial noise to alleviate the problem of signal-leak, improving inference quality. However, they suffer from limited diversity and cumbersome inference. At the same time, first-round inference of FreeInit (Wu et al., ) still exhibits a training-inference gap.

In contrast to existing methods, our approach utilizes images as the stepping stone for text-to-video generation. This novel pathway aims to produce visually-realistic and semantically-reasonable videos while maintaining manageable computational overheads, as detailed in Sec. 4.

Figure: Figure 8: Qualitative comparison. Each video is generated with the same text prompt and random seed for all methods. Our approach significantly improves the quality of the generated videos while showing excellent alignment with text prompts.
Refer to caption: x8.png

Figure: Figure 9: Impact of hyperparameters. For different texts, the optimal parameter settings are different, and the sensitivity to parameters also varies. However, they all significantly outperform the baseline. In this paper, we provide an empirical setting that is mild for most cases, serving as a performance lower bound for I4VGen. I4VGen supports fine-tuning parameters on a per-example, achieving higher-quality videos.
Refer to caption: x12.png

Figure: Figure 10: Failure cases. I4VGen is designed to fully unleash the potential of existing video diffusion models, but it still cannot synthesize high-quality videos that are out of the distribution. For example, the text marked in red.
Refer to caption: x13.png

## Appendix B Experiments

### B.1 Qualitative comparison

We provide more visualization results in Fig. 8, it can be seen that our method generates more semantically plausible and photo-realistic results than its counterparts. We provide the videos shown in the main paper and appendix in mp4 format in the Supplementary material.

### B.2 Quantitative comparison

On hyperparameters. I4VGen is a training-free method that improves video generation performance by correcting the inference process. It is obvious that I4VGen is also a case-wise method, where different cases correspond to different optimal hyperparameters. In this paper, we provide an empirical setting that is mild for most instances, serving as a performance lower bound for I4VGen, and facilitating large-scale quantitative comparisons. Furthermore, we also provide a visualization of the impact of hyperparameters in Fig. 9, which shows that carefully tuned hyperparameters can achieve higher-quality videos.

### B.3 Failure cases and discussions

We provide the failure cases in Fig. 10, I4VGen is designed to fully unleash the potential of existing video diffusion models, but it still fails to synthesize high-quality videos that are out of the distribution.

## Appendix C Code

We also provide the code for I4VGen in the Supplementary material.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - The integration with existing image-to-video diffusion models is interesting, but the authors are suggested to combined with more I2V models, especially several recent ones.
- More ablation studies are required to show whether the anchor image selection and the NI-VSDS  are optimal.
- In the bottom of Fig. 6, albeit better image quality, it seems that the motion of the proposed method is smaller than that by SparseCtrl. More experiments are suggested to assess this aspect.
soundness: 2
strengths: + Two-stage text-to-Video Generation method I4VGEN.
+ Can be integrated into existing image-to-video diffusion models.
summary: This paper presented a two-stage text-to-Video Generation method I4VGEN, i.e., anchor image synthesis and anchor image-augmented text-to-video synthesis. Experiments are provided to assess its effectiveness in text2video generation and enhancing I2V methods.
weaknesses: - The integration with existing image-to-video diffusion models is interesting, but the authors are suggested to combined with more I2V models, especially several recent ones.
- More ablation studies are required to show whether the anchor image selection and the NI-VSDS  are optimal.
- In the bottom of Fig. 6, albeit better image quality, it seems that the motion of the proposed method is smaller than that by SparseCtrl. More experiments are suggested to assess this aspect.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What are the effects of using different image generation models? Can better video quality be achieved if a better image model (e.g., FLUX) is used to generate anchor images?
2. Will the SDS degrade image quality? It would be better to provide quantitative results in Table 2 to show its effects.
3. In Fig. 8, the prompt for the last video is mistaken.
soundness: 2
strengths: 1. The proposed method uses a pre-trained image generation model to improve frame quality in text-to-video generation, which is helpful for high-quality video generation.
2. The presented results demonstrate good quality.
summary: This paper proposes a video diffusion inference pipeline that leverages image generation techniques to enhance a pre-trained text-to-video (T2V) diffusion model. Instead of directly generating videos from noise, the method first utilizes a text-to-image (T2I) model to generate a high-quality anchor image. This image is then used to produce an initial video via Score Distillation Sampling (SDS) through the T2V model. A regeneration process is adopted to refine it, resulting in the final video. Experiments have been conducted to evaluate the effectiveness of the proposed method both qualitatively and quantitatively.
weaknesses: 1. The proposed method appears to integrate the T2I model with SDS distillation for video generation, and the contribution seems incremental.
2. The motion observed in Fig. 1 appears to be smaller compared to the baselines.
3. There is a lack of analysis for different regeneration steps.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Concerns about inference time. The proposed method consists of two stages: anchor image synthesis and anchor image-augmented video synthesis. The reviewer wants to know whether the time in the table includes the time used in the first stage.
2. Low-quality face video in Fig.6. In 1st row of Fig6, SparseCtrl produces a video in extremely low quality, could the authors explain reasons?
soundness: 3
strengths: 1. This paper introduces a training-free pipeline called I4VGen to improve the performance of text-to-video diffusion models throught image reference information.
2. A simple yet effective generation-selection strategy is proposed to obtain high-quality-images, while a noise-invariant video score distillation sampling is introduced for image animation.
3. Extensive experiments show that the proposed method comsiderably outperforms the performance of video diffusion baselines in terms of video quliaty.
summary: This paper presents I4VGEN, a novel video diffusion inference pipeline that enhances pre-trained text-to-video models without additional training. It tackles the complexities of spatio-temporal modeling by utilizing advanced image techniques in two stages: first, synthesizing anchor images with a strategy to ensure visual realism and semantic accuracy; second, augmenting these images to generate videos through a noise-invariant video score distillation sampling (NI-VSDS) method. This process also includes a video regeneration step for refinement. Experiments show that I4VGEN significantly improves the visual quality and textual accuracy of generated videos and can be easily integrated into existing models, boosting overall video quality.
weaknesses: 1. The technical contributions of the paper are somewhat limited. The proposed noise-invariant video score distillation only modifies some hyper-parameters of the original SDS techinque.
2. Compared to the baseline results, the video actions enhanced using the proposed method in this paper are minimal or essentially stationary. The metrics in Table 1 also show that the proposed method heavily harm the dynamic degree of generated videos.
3. AnimateDiff relies on high-quality LoRAs to improve the quality and consistency of generated videos. Please provide generated videos of AnimateDiff with high-quality LoRAs for a fair comparison.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: I have several concerns as follow
1. More proof of the anchor image synthesis. Generally speaking, anchor image act as a low frequency component to mitigate the information leak. A question then rise that whether the anchor image synthesis is necessary. In ablation study, the authors use an example in Fig.5 to validate the assumption. Meanwhile, the authors also confirms that without the generation-selection strategy, the proposed method still performs well. Can the authors give more examples to support this claim?
2. In User study, the 20 volunteers with expertise in image and video processing participated. Does the expert bias exist? that is to say, the participants are more tolerant to the defects of the methods? Will the ordinary participants from a consumer's perspective be better?
3. It's a trend that video generation tends to produce more diverse content with long duration, for example, the large camera pose change. Will the proposed method still work?
4. The proposed method further increase the inference cost. More comprehensive measures are better presented from a practical perspective.
soundness: 3
strengths: The paper focuses on the video generation quality from a noise initialization viewpoint, which is hot and also vital to AI-generated content. The authors try to use the off-the-shelf T2I and T2V models in a novel way, to be specific, anchor image synthesis using T2I and motion prior using T2V in a novel SDS way.
summary: The paper proposes a novel method to initialize the noise for T2V. To endow the noise with motion prior, the paper take the idea of SDS in T-to-3D in a video generation way, which is considered to be novel and inspiring to the community.
weaknesses: Generally speaking, video generation is notoriously for its lengthy inference cost. Noise initialization adds more cost, which is not practical. The authors are thus encouraged to add more discussion. Besides, the experiments can be more consolidated to enhance the authors' claim. Please refer to the question part for details.

Meta-review: While the proposed I4VGen method offers practical improvements for text-to-video generation, concerns about its novelty (dmyx), incremental technical contributions (hdjr) and practicality (tiD5) remain. Despite efforts in the rebuttal to address these issues, some reviewers feel that the contributions, particularly NI-VSDS, are minor modifications of SDS. In addition, the trade-off between dynamic motion and temporal consistency raises questions about the method's general applicability. Given the borderline scores and unresolved concerns, the decision is to reject. The authors are encouraged to refine the contributions and address the reviewers’ feedback for future submissions.

Decision: Reject