Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model
Authors: Min Zhao
Hongzhou Zhu
Chendong Xiang
Kaiwen Zheng
Chongxuan Li
Jun Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model
image-to-video diffusion model
video generation
video diffusion model
Primary area: diffusion_based_models

Abstract: Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.

Identifying and Solving Conditional Image Leakage in Image-to-Vide Video Diffusion Model
Min Zhao!.3 Hongzhou Zhu 1.3 Chendong Xiang Kaiwen Zheng
Chongxuan Li Jun Zhu 1,3.4
Dept. of Comp. Sci. & Tech., BNRist Center, THU-Boste ML Center, Tsinghua University Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China
Shu, Beijing, China; 'Pazhou Laboratory (Huangpu), Guangzhou, China (gracezhao1997, xiangxyaw, , zww.zuw (ighall..mmm..... com; zhuhz220mails. ts ininguua. edu. cn; chongxuan] iillliirrr @@ruc. edu. cn; dcazj Stsigguaing edu. cn
Abstract
Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-videss viieo diffusion models (12V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from earlier time step to avoid the unreliable large-time steps of V-M-,,s, as well as an initial noise distribution with optimal analytic expressions (Analytic- by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training. applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency various I2V-DMs
it. We validate these general strategies open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: https:/ ctt////oddii//// h/c//oiiddimmeeeeee leak. github.io/
collected
Introduction
Image-to-videe (I2V) generation aims to generate videos with dynamic and natural motion while maintaining the content of the given image. It allows users to guide video creation from the input image (and optional text), thus increasing controllability and flexibility in content creation. Like
the remarkable progress in text-to-image (T2I) generation [42 39 21
[8 5) and text-to-video [1, diffusion models have also obtained promising results for I2V
8
(T2V) generation 110 46
generation 9 15
63
69
70) However, such models are not fully understood.
In this paper, we observe that existing image-to-vides diffusion models (12V-DMs) tend generate videos with less motion than expected (see Fig. [ We attribute this to previously overlooked issue called conditional image leakage (see Sec. 3.I. Normally, the noisy input contains the motion information of the target video, and 12V-DMs should rely on it to predict motion, while the static conditional image provides content guidance. However, practice, as the diffusion process
Equal contribution.
*Coressondence to: C. Li and J. Zhu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Initial noise
Output MS ISp-npptt MS against Input MS
Output MS Input MS
Output video
12V-DM sampler
Conditional image
-5
Compare
-10
SVD
Ours
-15
5
10
20
40
80
100
127
Input MS
Input MS
Figure 1: The issue of existing I2V-DMs. Regardless of input motion scores (Input MS). the output motion scores (Output MS) are consistently lower than expected. In contrast, our method yields output motion scores either higher or lower than Input MS with reduced error.
progresses especially at large time steps, the noisy input becomes heavily corrupted, while the conditional image preserves extensive detail of the target video. This biases the model to over-rely on the conditional image and neglect the noisy input, leading to videos with reduced motion. To validate this, we corrupt the ground truth (GT) clean video and compute one-step clean video predictions at each time step. As shown in Fig. 2 the predicted clean videos exhibit markedly reduced motion than GT at large time steps, indicating leakage.
Based on the above analysis, we attempt to address the challenge from both inference and training aspects (see Sec. 3.2). First, we present simple yet effective inference strategy that starts the video generation process from an earlier time step, thus avoiding the unreliable large-time steps of the 12V-DMs. To further enhance performance, we derive an initial noise distribution with optimal analytic expressions (Analytic-lo) Init) by minimizing the KL divergence between and the true marginal distribution to bridge the training-inference gap. Second, to mitigate leakage during training, we propose time-dependent noise distribution (TimeNoise) that increases noise levels at larger time steps, effectively disrupting the conditional image and reducing model dependency on it. This is achieved by employing a logit-normal distribution with a center that gradually shifts over time. Finally, our method achieves higher motion scores with reduced motion score error and ensures that the predicted clean video maintains motion dynamics comparable to the ground truth across all time steps, effectively mitigating conditional image leakage. Notably, our general strategies adaptable and VE-SDE 9
to various 12V-DMs based on both VP-SDE 12 63 70
framework.
using our collected open- domain images (ImageBench) and the UCF101 dataset. We conduct a user study on ImageBench and report FVD [55] IS 77. and motion score 9 on UCF101. For motion-condittoned models [9] 77), we also report the motion score error between the generated video and the input motion score at different levels. Extensive experimental results demonstrate that strategies outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control.
Empirically, we validate our methods on various 12V-DMs 9
63 12
70
2 Background
Diffusion Models. Diffusion models gradually perturb the data 2g ~ q((a) via a forward diffusion
process and reverse the process to recover it. The forward transitional kernel 4:jo (#))))) is given by
2t 0g + OiC, C c~ (0, t [0,
(1)
where 04 and de are the noise schedule chosen ensure that ær contains minimal information about Zp. Such forward diffusion processes can be viewed as stochastic differential equations (SDEs), among which two prevalent types are commonly used [48 29 One the variance- preseeriig SDE (VP-SDE) [2] [24, where ot + of = 1 with a, - 0 as / T, ensuring pr (x() N VaT:; 0, T). The other the variance exploding SDE (VE-SDE), where o, and or set to large constant, resulting in pr (ΦT) s W( 0, T). Such models can be parameterized with predictinn
2
model ca (zz,, t) (e-prediction) 24 .and the parameters are learned by minimizing:
E (=a [2]co(x,,)) f) -c1|3]

---

time step t
7
As increases, noisy imput loses GT detail, but conditional image retains
Noisy input
Conditional image
Over-reliance
Over-reliunce
diffiusion model
12V diffusion model
12V diffusion model
Clean video prediction
GT motion
GT motion
< GT motion
Conditional image leakage
Conditional image leakage
Figure 2: Identifying conditional image leakage. As time step progresses, the noisy input becomes heavily corrupted, whereas the conditional image retains considerable detail from GT. This biases the model to over-rely the conditional image at large I, resulting in videos with less motion than GT.
where the noisy input
2rjo (o)) Alternative parametrizations such 2p zu-prediction 29 o-prediction [45]. F- f'-prddiction [28] are also commonly applied. The e-prediction and zo-prediction aims to predict the added noise or clean video from noisy input æy. Starting from 2r P1 (ær).
various samplers 47 B3 34 28
can be employed to generate data. More recent studies [66] further
demonstrate that diffusion models can generate realistic images with controllable semantics given only a few labels.
Diffusion Models for Image-to- Video Generation. Given an image yo from the open domain, goal of 12V to generate video Xo = {ar with dynamic and natural motion while keeping alignment with the appearance of yo- This task can be formulated as designing conditional distribution po PX Nolyo), which achieved by a conditional diffusion model minimizing:
Exo.vo.e.t [Xc, yo. -
Typically, Yo is the first frame of Xo and DynamiCrafter [63] adopts randomly selected frame from Xn as yo- The key issue is to effectively integrate the conditional image Yo into the diffusion model. Most methods use CLIP image embeddings [40] maintain the semantic content of Notably, VideoCrafterl [12] and Dynamicrafter [63] employ the last layer's full patch visual tokens from the CLIP ViT, enriching the encoded information, and other approaches prefer the class token layer. Yet, solely depending on these embeddings, such as in VideoCrafterl [12), compromises detail retention, resulting in degradation of the image alignment. To enhance detail representation, 12VGen-XL [69 combines the conditional image with the noisy initial frame, while VideoComposer [57] develops STC-encoder for multiple conditions. Although superior to CLIP image embeddings. these strategies still fail to fully retain the conditional image content. To mitigate this, AnihateAnything [15). Dynamicrafter (63) and SVD ( directly concatenate noisy video X, with Yo, which injects detailed information to the model. Apart from the prior work mentioned before, we discuss other related work about diffusion models for image generation and video generation in the Appendix
where X,
3 Method
Although existing 12V-DMs discussed in Sec. have achieved significant progress, such models are not fully understood. this section, we first identify a critical yet previously overlooked issue in
12V-DMs: conditional image leakage (CIL.) (see Sec. 3.1). We then address this issue from both inference and training aspects accordingly (see Sec.
Finally, we offer insights into existing
12V-DMs through the lens of CIL (see Sec. 3.3

---

start time M
0.87
increase motion
SVD Analytic-Init
Output IIputt MS
Naive initialization
-5
SVD
-10
+Analytic-nni
10
80
100
127
Imput MS
better quality
PIA vs + Analytic-Init
Conditional Image
0.01
Output MS Inpur MS
Analytic-In Init
-0.02
PIA +Analytic-Init
-0.03
-0.04
0.172
0.221
0.344
Input MS
(a) Impact of tuning M and Analytic-Init on visual quality.
(b) Impact of Analytic-Init on the motion score.
Figure 3: Benefits of Analytic-Init. (a) An early start time enhances motion but a too-small M degrades visual quality due the training-inferent gap, which Analytic-Inil helps to reduce. (b) Analytic-Init produces higher motion scores with lower errors, mitigating conditional image leakage.
3.1 Identifying Conditional Image Leakage in Image-to-video Diffusion Models
we observe that, regardless of the input motion scores, the motion scores of generated videos from existing 12V-DMs [9] 770 consistently lower than expected. This raises the question: why do these models always produce lower motion scores, rather than fluctuating above or below the expected values, as observed in our method?
As shown Fig. 1
To understand this, we need to consider the source of motion information in the generated videos, which comes from the noisy imput X₃. Ideally, I2V-DMs should rely primarily on X₁ for motion, with the static conditional image yo providing content guidance. However, as shown in Fig. 2 at large time steps, the noisy input X, becomes increasingly corrupted, while the conditional image Yo retains significant information of the target video. This biases the model to over-rely on the conditional image and neglect the noisy inputs, leading videos with reduced motion.
To validate this, we corrupt ground truth (GT) clean video X, via the forward transition kernel in Eq. I and use it as the noisy input to compute the one-step prediction X at time
X.- = (Xe (X:- o t)) /az.
(4)
Ideally, X should predict the GT Xo from noisy input X, and exhibit comparable motion dynamics. However, as shown in Fig. as time exhibits markedly reduced motion than GT, indicating the conditional image leakage. This results in videos with reduced motion starting from time T., Notably, recent techniques that adjust the noise schedule towards higher noise levels [4 27 for details).
at large time steps, X
32 may further exacerbate this issue (see Appendix
3.2 Solving Conditional Image Leakage in Image-to-video Diffusion Models
Building upon the above analysis, this section presents general strategies to address the issue of conditional image leakage in both the inference and training aspects.

---

a
0.2
Probability Density
0.5
5
0.5
m(()
0
-0.5
0
0.28m
0.43m
0.63m
0.8,3m
3m
0.27
0.47
0.6T
0.87
T
3.
(a) TimeNoise Pe (5,) with hyperparameters 8m (maximum noise) and p(t) (distribution center).
(b) Distribution center ju(t) = 26 with
controlling monotonic behavior flexihly.
Baseline
a =
5
a=
a
0.2
3m
25 8m = 100
8m 700
Conditional Image
more motion, lower image alignment more motion, lower image alignment
(c) Effects of tuning in (() and [m, with higher noise levels enhancing motion but reducing alignment.
Figure 4: Visualization of Time and the impact of tuning its hyperparameters. (a) The designed Pa (3.) favors high noise levels at large t, gradually shifting to lower noise levels as decreases. This is achieved by (b) jut) increasing monotonically with t. Finally, (c) modifying and 3m enables a trade-off between dynamic motion and image alignment.
Inference strategy. As discussed in Sec. 3.1 conditional image leakage easily occurs at large time steps. To this end, straightforward solution is start the generation process from an earlier time step . E (0, T'), thus avoiding the unreliable later stages of I2V-DMs. Let pM (XM) denote
the initial noise distribution at the start time M. Initially, we set PM
PM (XM) ) (Xr). i.c. NN,,
in VP-SDE 24 63 12 70 or (O, 1) in VE-SDE (29 D. As illustrated in Fig. 3 (a), this
straightforward strategy markedly improves motion dynamics without sacrificing other performance.
However, smaller M value (e.g.. M 0.87) results in poor visual quality due the training- inference discrepancy.
To mitigate this gap, we propose Analytic Noise Initialization (Analytic-Init) to refine the initial noise distribution par (XM) by minimizing the KI. divergence between and the true marginal distribution qM (XM) of the forward diffusion process. Inspired by previous work [4] 3 64], we demonstrate that when PM M)) is modeled as normal distribution. N ((X,,, Mi ppp of I). the optimal mean μ and variance o have analytical solutions, as stated in Proposition
Proposition 1. Given a normal distribution Pas(XN) = N(X Hp. o²I) and qas (X is the margin distribution of diffusion forward process at time M. with the forward trainsition kernel X (x) min D DKL (gas (X (XM)) yields the following optimal solution: pp.03
N(X aM I2 the minimization problem
(ar(X()) d
, = as ax Xol. = a.y
+ +
(5)
5

---

SVD-naive-tune vs SVD-TimeNoise
GT
Output MS Input MS
Clean vidco prediction
-15
-30
GT motion
t. Slan lime
SVD-naive-tume
-60
NVD-TimeNoise
z GT motion
-75
5
10
20
40
80
100
127
Input MS
GT motion
No conditional image leakage
(a) One-step elean vidco prediction TimeNoise.
(b) Impact TimeNoise on the motion score.
Figure 5: Benefits of TimeNoise. TimeNoise (a) generates Xy- that maintains motion dynamics comparable to the GT across all time steps, and (b) achieves higher motion scores with lower errors, effectively reducing conditional image leakage.
where q(Xo) denotes the data distribution, d denotes the dimension of the data, and X denotes the j-th component of Xo.
The proof of Proposition is provided in Appendix A Empirically. fe," and 2² can be estimated using the method of moments The steps for inference outlined in Algorithm As demonstrated by the qualitative results in Fig. 3 (a) and quantitative results in Tab. [8] Analytic-Init improves video quality by reducing the mraining-infereee (b) and Tab. M Analytic-Ini produces higher motion scores with lower errors, allowing for more accurate motion control and reducing conditional image leakage.
gap, especially for smaller M. Finally, as shown in Fig. 3
Training strategy. In this section, we show how to address the issue of conditional image leakage during the training phase. As outlined in Sec. 3.T the conditional image yo retains substantial details of the target video, causing 12V-DMs to rely heavily on it. To mitigate this, a natural approach is to perturb yo relieve this dependency. Our first attempt is introduce noise that in X,, aiming to balance the model's challenge of predicting clean video from X, or yo. thereby lessening reliance yo- However, this strategy also makes it difficult employ yo. resulting in lower video quality.
similar scale to
To overcome this, propose noise distribution yo that introduces substantial noise to prevent leakage while maintaining a cleaner yo to aid content generation. Given that X, contains less information about Xo as time progresses, increasing the risk of leakage, further develop time- dependent noise distribution (f,) (TimeNoise). The key principle is to favor high noise levels large time steps to sufficiently disrupt yo. shifting towards lower noise levels the time step decreases. To achieve this, we employ a logit-normal distribution 18 defined as below:
Bm
1
Bm)) = 2T 8.(3m B.)
(6)
where logit( (...) follows normal distribution centered around jut) with standard deviation of 1. This noise distribution includes two hyperparameters: 3... the maximum noise level, and jut), the center of the distribution. As illustrated in Fig. 4 (a), we can adjust ju(t) over time t to satisfy the previously mentioned design principle. Finally, the noisy conditional image y, at time t obtained by y, Yo + B.c, where B, ~ (B.), e (N I). We also tried other adding noise choices but found them to less effective (see Appendix DD. During inference, we a fixed noise level conditional image across all time steps, following CDM(25)
the model is trained with varying noise levels. Empirically, we find directly using clean conditional image performs well and thus adopt it for simplicity. The full algorithm is shown in Algorithm? 2

---

Conditional image
12 12VGen-XL
VidcoCrafterl
Ours
Conditional image
SVD
Ours
rightward pan camera movement
steam rising from the tcapot
upward pan camera movement
the person jumping
Figure 6: Understanding exiting work from conditional image leakage. 12VGen-XI [69 and VideoCrafter1 [12] mitigates the leakage the expense image alignment. The SVD produces videos with camera movements while keeping objects relatively static to meet high motion scores, while ours generates videos that feature both natural and dynamic object movements.
Next, we conduct systematic analysis of the two hyperparameters to investigate their impact on video generation. Firstly, j(t) designed to increase monotonically with time, ranging from (0) =-1 to p(())= = 1. To formalize this, we define μ(() as power function: μ(() = 2/ 1, where > 0. This formulation allows flexible tuning of a to control the monotonie behavior, where smaller values of a cause higher noise levels to be sampled at later time steps. As shown in Fig. (b). (1)
corresponds to linear increase; (2) a E (0, 1) represents a concave function, indicating a faster noise level increase; and (3) 1 corresponds
convex function, indicating slower increase in noise levels over time. As shown Fig. 4 (c), higher noise levels (e.g., when a < 1) lead to increased dynamic motion but reduced temporal consistency and image alignment. For the maximum noise level Bm. the only constraint is that must be greater than 0. As shown in Fig. (c). a higher 8m enhances dynamic motion but decreases temporal consistency and image alignment, while lower 3m reduces motion. Additionally, we apply TimeNoise to the CLIP Image Embedding for both VideoCrafter1 [12) and DynamiCrafter (63). as they use full patch visual tokens from CLIP, which contain substantial information about the conditional image, increasing the likelihood of leakage.
Finally, we replicate the experiments described in Sec. 3.1 with results presented in Fig. 5 Tab. These results demonstrate that TimeNoise achieves higher motion scores with reduced error and ensures that X,-0 maintains motion dynamics comparable to the ground truth across all time steps, effectively mitigating conditional image leakage.
[
3.3 Understanding Existing Work from Conditional Image Leakage
In this section, we analyze popular I2V-DMs 63
through the lens of CIL. Although these models do not explicitly address this issue, we believe their strategies mitigate it to some degree.
22
Firstly, some methods only use partial information from the conditional image, which can help reduce leakage. For example, VideoCrafterl 12) only utilizes CLIP Image Embedding. and 12VGEN- XL [69] adds yo to the first frame of the noisy input. However, shown in Fig. 6 the videos generated by these methods do not fully capture the details of yo- To address this, models like Dynamicrafter [63] directly incorporate Yo into I2V-DMs, improving detail preservation but also increasing the risk of leakage. Dynamicrafter selectively refines spatial layers while preserving the pre-trained temporal layers, which contain motion priors and thus maintain motion dynamics certain extent. However, it does not inherently solve the leakage issue. Moreover, some methods introduce external signals 70 65| 62 60), forcing the model align with additional conditions, which reduces its dependence on and helps mitigate leakage. However, we argue that they do not address the core challenge of 12V-DMs, which should predict clean video primarily from noisy input to capture motion information. As illustrated in Fig. 6 the SVD often results in static objects with excessive camera movements to meet high motion score requirements. In contrast, our method generates videos with natural, vivid object movements. In summary, while the above methods mitigate
7

---

Conditional image
Bascline
Analytic-Init
VC
A cartoon girl with brown curly hair splashes joyfully in bubble-filled bathtub
DC.
orange being half and splashing into liquid, likely milk, swinging fast.
SVD
Conditional image
Deseline-naive-ie
Toise
VC.
sporkler being on ground.
DC
cat lying on the bed.
SVD
Figure 7: Qualitative results of TimeNoise and Analytic-Init applied to various I2V-DMs. Ours significantly enhances video dynamism while maintaining image alignment and temporal consistency. VC. and DC. denote VideoCrafterl 12 and DynamiCrafter [63] respectively.
CIL some extent, our approach provides a more effective solution to the fundamental challenges in 12V-DMs, enabling more precise motion control and enhancing naturalness by focusing more on the noisy input.
4 Experiments
4.1 Setup
Datasets. We WebVid-2M [2] as the training dataset, with all videos resized and center-cropped
to 320 x 512 at 16 frames and 3 fps. For evaluation, we use UCF101 [49] and our ImageBench
dataset, which includes diverse categories (e.g., nature, humans, animals, plants, food, vehicles) and
complex elements like numerals, colors, and intricate scenes, similar to DrawBench 44 total, 100
images collected from various websites and T2I models such SDXL 39 and UniDiffuser (5)
Evaluation Metrics. On UCFI01, we report Fréchet Video Distance (FVD), Inception Score (IS), and Motion Score (MS). For ImageBench, conduct studies with 10 subjects perform pairwise comparisons of our methods against baselines, evaluating motion, temporal consistency, image alignment, and overall performance. For motion--ooditioned methods, motion score eттоr between the generated video and the input motion score at various levels. Motion scores are computed using flow maps following SVD [ except for the PLA [70], where follow its original algorithm and compute the L1 distance. More details on the metric computations are provided in Appendix
also report the
Implementation Details. For Analytic-Init, we use 5000 samples from the Webvid-2M dataset to estimate Pwr( (Xm) by default. We set M 0.967, 0.967 and σ 100 for VideoCrafterl [[2] DynamiCrafter [63]. and SVD [9) on UCF101. For ImageBench, we adjust M to 0.92T, 0.92T and αM 100. For the TimeNoise, we set 9m 25, 100, and 100 for VideoCrafterl. DynamiCrafter, and SVD, respectively. The function p(()= = 5 1 is applied across all baselines. These baselines are fine-tuned for 20,000 iterations, using either the official code replication of the official settings except for batch size. More detailed information can be found in the Appendix

---

Table 1: Quantitative results of TimeNoise and Analytic-Init applied various I2V-DMs on the UCF101 dataset. AMethod>-naive tune represents naively fine-tuned baseline using the same training setup as our TimeNoise, ensuring a fair comparison. <Methods-CII. denotes the full version using both TimeNoise and Analytic-Init.
Model
FVDJ
1st
Motion Score f
DymiCrafter [33
363.8
16.39
50.96
DymiCrafter + Analytic-Init
316.3
17.66
71.04
DymiCrafter-naive-ele
382.5
21.12
31.68
DymiCCrffer--nivve-unn Analytic-Init
342.9
22.71
50.08
DymiCrafter- Noie
334.9
21.42
72.32
DymiCrafter-CII
332.1
22.84
73.92
VideoCrafterl [22
353.9
18.75
63.36
VideoCrafterl + Analytic-Init
341.6
19.86
139.04
VideoCrafterl-naiiee
460.3
23.98
62.72
VideoCrafter -aive tune + Analytic-Init
450.1
24.50
65.12
VideoCrafter1-Timeel
452.2
24.62
64.80
VideoCrafterI-CII
443.7
25.11
66.7
SVD 9
388.3
36.32
16.64
SVD+ + Analytic-Init
382.0
36.81
19.68
S SVD-naive-tune
311.0
22.03
9.60
SVD- -VD-naive-tune Analytic-Init
277.1
22.18
20.64
SVD-TimeNoise
272.2
23.01
20.96
SVD-CII
272.4
25.18
21.44
4.2 Results
The effectiveness of our inference and training strategy. We validate strategies on following 12V-DMs: VideoCrafteri [2]. DynamiCrafter 63) and SVD (54) We validate our inference strategy against the original method and naively finetuned version, and our training strategy against a naively finetuned version. The quantitative comparisons and qualitative results are presented in Tab. [
Fig. 3 Fig. 5 and Fig. 7 leading to several key observations. First, our strategies significantly improve motion scores and reduce motion score error, demonstrating their effectiveness in precisely controlling motion degrees and mitigating conditional image leakage. Second, we achieve both dynamic motion and high video quality, as reflected by improved FVD and IS scores. The user study in Fig. 88 further supports that our strategies enhance video dynamism while preserving image alignment and temporal consistency, yielding superior overall performance.
Evaluating the combined inference and training strategies. In this section, we aim to investigate the necessity of combining two strategies. As illustrated in Tab. 7 for DynamiCrafter (63 and SVD 19], our TimeNoise effectively mitigates conditional image leakage. rendering the additional inference strategy less impactful. Conversely, for VideoCrafter [12], which relies solely on CLIP image embedding for information, employing excessive TimeNoise disrupts image alignment. Hence, we utilize moderate TimeNoise (3m 25). where the inference strategy remains effective.
Comparison with SOTA noise initialization and conditioning augmentation methods. We compare our Analytic-Init with SOTA noise initialization methods [4]
61] 61] 19 and our TimeNoise with SOTA conditioning augmentation (25]. As shown in Tab. 2 Analytic-Ini Init outperforms all
baselines. It achieves higher motion scores than FrameInit [41], which limits motion by duplicating the conditional image static guidance. Compared to FrecInit [61. our method reduces inference time by more than half while slightly improving performance, as Freelnit's iterative process increases time. Compared Progressive Noise [19), Analytic- performs better by generating videos from an earlier time step, a gap the baseline cannot handle effectively. As shown in Fig. in the Appendix, CDM 25 results in lower motion, whereas method produces more dynamic videos. Additionally,
naive baseline using single value for 3, = 3m p(+)+1 leads to poor image alignment, further demonstrating the effectiveness of our time-dependen noise distribution.
2

---

DC
DC:
DCNT
DC-NT.
55505
5.00%
VC.
VC.
bolllttt---aa Ini
VO-NT. 24.00%
VCNT.
$3.00%%
SVD-NT
SVD-NT
DC-NT
50.000
DC-NT
30%
TimeNobe CIL
VC-NT.
34.005
05.10%
VC-NT
SVD-NT.
SVD-NT
3 SUS
DC-NT.
DCNNT
VC-NT.
VC-TT
SVD-NT
90.05
SVI-NT
25.00%
50.9
75.0%
1000%
50000% Dynamic motion
75,005
Overall
DC-NT.
DC-NT.
Amalytä-Ini Init
VC.
VC
VC-NT.
VC-NT.
35.00%
42000%
SVD
10.0%%
60%
SVD
11.005
90%
SVD-NT.
50300
40.00%
SVD-NT,
$2.005
40000%
DC-NT
45.90%
DC-NT
XL.00%
Time. CTL
VC-NT.
VC-NT.
10.30%
SVD-NT.
SVD-NT.
20,00%
DC-NT.
VC-NT.
VC-TT.
SVD-NT.
SVD-NT.
Image alignment
Temporal consistency
Baveline
Amalytic-nnit
TimeNoise
CIL (full version)
Figure 8: User preference percentages for ours against baselines. <Method>-NT. is the naively fine-tuned baseline. DC. and VC. denote DynamiCrafter and VideoCrafterl. 1. Our strategies signifi- cantly enhance video dynamism while maintaining image alignment and temporal consistency, thus achieving superior results overall.
Table 2: Comparison with SOTA noise initialization methods. Rate. is the average user ranking of each method based on overall performance. Time measures the duration needed generate a video.
Method
FVD1
IS
MS
Rate. J
Time
FrameInit 41
380.7
20.09
32.16
4.57
24.3s/it
FreeInit [61]
347.4
22.66
46.24
1.95
49.6s/it
Progressive Noise 19
358.1
21.35
49.76
3.31
23.3s/it
Analytic-Init
342.9
22.71
50.08
1.77
22.6s/it
Conclusions and Discussions
In this paper, we identify common issue in I2V-DMs: conditional image leakage. We address this challenge from two aspects. First, we introduce an inference strategy that starts the generation process from an earlier time step to avoid the unreliable late-time steps of I2V-DMs. Second, design time-dependent noise distribution for the conditional image to mitigate conditional image leakage during training. We validate the effectiveness of these strategies across various 12V-DMs.
Limitations and broader impact. One limitation of this paper is the need to balance TimeNoise to prevent conditional image leakage while maintaining image integrity. While we demonstrate the effectiveness of our strategy on existing 12V-DMs, we do not provide a definitive choice for scratch-trained model. We leave this in the future work. Furthermore, we must the method responsibly to prevent any negative social impacts, such as the creation of misleading fake videos.
Acknowledgement
This work was supported by NSFC Projects (Nos. 62350080, 62106122, 92248303, 92370124, 62350080, 62276149, U2341228, 62076147), Beijing NSF (No. L247030), Beijing Nova Program (No. 20230484416), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize.
10

---

References
[1] Jhon Atchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-272, 1980.
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[3] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covari- ance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:206.007309 2022.
[4] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06500 2022.
[5] Fan Bao, Shen Nic, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint 2rXxi:2333.0655. 2023.
[6] Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang. and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.042. 2024.
[7] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.0199 2018.
[8] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. htlps://cdn. openai. com/papers/dall-e-3 pdf. 2(3):8, 2023.
[9] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[10] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-22575. 2023.
[11] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman. Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[12] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023.
[13] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047 09047,, 2024.
[14] Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint wrX1::2011.09972 2023.
[15] Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine- sine-grained open domain image animation with motion guidance. arXiv e-prints, pages arXiv 2023.
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
[17] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[18] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprint arXiv:2403.033: 003.000006 2024.
11

---

[19] Songwei Ge. Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang. Ming-Yi Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 22930-22941, 2023.
[20] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin. Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arX1v:2311.10709 2023.
[21] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 2023.
[22] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01666, 2022.
[23] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230: 2022.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, Advances in neural information processing systems, 33:6840-6851, 2020.
[25] Jonathan Ho. Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[26] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.1586. 2022.
[27] Emiel Hoogehoom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213- 13232. PMLR, 2023.
[28] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffus generative models. Advances in Neural Information Processing Systems, 35:26565-2657 2022.
[29] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the clbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
[30] Gihyun Kwon and Jong Chul Ye. Diffusion- image translation using disentangled style and content representation. arXiv preprint arXiv:2209. 1 15264, 2022.
[31] Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding. and Jingdong Wang. Videogen: A inference-guidee latent diffusion approach for high definition text-to-vide video generation. arXiv preprint arXiv:2309. 00398, 2023.
[32] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5404 5411, 2024.
[33] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787. 2022.
[34] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[35] Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding, Vdt: General-purpose video diffusion transformers via mask modeling. In The Twelfth International Conference Learning Representations, 2023.
[36] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.00 2024.
12

---

[37] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
[38] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[39] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.019552 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[41] Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04322 2024.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks Medical image computing and computer-assister
for biomedical image segmentation. interrention-MIICAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer. 2015.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic rext-to-image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-3666 2022.
[45] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprini arXiv: 00512, 2022.
[46] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint Stt1:22009114992 2022.
[47] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 1.XI::2010002502, 2020.
[48] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[49] K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402 2012.
[50] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Aision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pages 402-419. Springer, 2020.
[51] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEFE international conference on computer vision, pages 1489-4497 2015.
[52] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[53] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[54] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint 23xiv:2308.06577 2023.
[55] Ting-Chur Wang, Ming- Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprint 1rXiv:1808.06600 06601, 2018.
13

---

[56] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention spatiotemporal diffusions for rext-to-video generation. arXiv preprint arXiv:2305. 10874, 2023.
[57] Xiang Wang, Hangjee Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang. Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024.
[58] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[59] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan
Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.0500 2024.
[60] Zhouxia Wang, Ziyang Yuan, Xintao Wang. Yaowei Li. Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[61] Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiv:2312.07537 2023.
[62] Wejia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefci He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. arXiv preprint arXiv:2403. 074200, 2024.
[63] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.121199 2023.
[64] Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, Jun Zhou, and Chongxuan Li. Unifying bayesian flow networks and diffusion models through stochastic differential equations. arXiv preprint arXiv:2404.155 15766, 2024.
[65] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine- eeneggranedd control in video generation by integrating text, image, and trajectory. arXiv preprint 2rXiv::338..08000 2023.
[66] Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu. Diffusion models and semi-supervised learners benefit mutually with few labels. Advances in Neural Information Processing Systems, 36, 2024.
[67] David Junhao Zhang, Jay Zhangjie Wu, Kai Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show- 1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint vrXiv:2309.18816 2023.
[68] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[69] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.041455 2023.
[70] Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-ande and-play modules in text-to-image models. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 7747-7756. 2024.
[71] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
[72] Ruowen Zhao, Zhengyi Wang, Yikai Wang, Zihan Zhou, and Jun Zhu. Flexidreamer: Single image-to-3e 3d generation with flexicubes. arXiv preprint arXiv:2404.00900 2024.
14

---

A Proof of Proposition I
According to Lemma 2 of Analytic-DPM 44, the KL divergence between initial noise distribution Par (Xm) and the actual marginal distribution qM (XM) can be expressed as:
DKI(yma (a (Xx)) = Dr. (N Mey [.) (X of + Mi Mg: [g)) H (qM((M))
(7)
where Mag Eng denote expectation and covariance matrix of qar qarggXXXx M). and H() denotes the entropy of distribution. Since qarl (Xr(Xx) Ar) is determined by the forward diffusion process and is independent of the 2, and of in PM Mx). the last two terms of Eq. can be considered as constants. Therefore, the optimization objective can be formulated as:
min (XM))M(MM) Mo
+ min (KL(N((MMii) Mg: Kyi
Up. of
(8)
According to the property of the KL divergence between normal distributions. (NI(N(MM: Mg: (X) Up, of.)) could be expanded as:
1 1 2 o7 up uyll 2 + (og)()))) + oF
tr(E,) log dot(Eg) d),
(9)
where d denotes the dimension of the data. Minimizing Eq. (9) yields:
" = 24 Eys (X (Xa))
(10)
Taking the derivative Eq.
with respect to of the optimal of comes to:
tr(Cou (XM]) + 1,, My
(11)
(X)))
(12)
Σd (Vor(()())))
(13)
We now represent (i, and oz" with Xg- Taking the Xar defined in Eq. (1) into Eq. 10 the optimal Hy could be further represented as
" a a (x Xo) Eggej(e),
(14)
aa =M [Xo]
(15)
Similarly, for the optimal variance 22 derive
Var(X M Var(am ao
(16)
Given that Xo and e are independent of each other, the variance Var(X (X) posed as:
can be further decom-
Var(X (1) Var(ans x(()) Var(one
(17)
Var(X o.
(18)
Finally, taking Eq. 18 into Eq.
13 the optimal o3 can be represented as
oet = a EE (Va((X)) + o
(19)
d
B
Implementation Details
B.1 Code Used and License
We validate our strategies on DynamiCrafter 63 (320 512 version), VideoCrafterl 12 version), and SVD [9] All used codes in this paper and their licenses are listed in Tab.
(320 x 512
15

---

Table 3: Code Links and Licenses.
Method
Link
License
VideoCrafterl 12
https://github.com http::/ggttttt.ccm//laaaaaaaaaaa
Apache License
DynamiCrafter
Apache License
SVD
models
MIT License
UniDiffusers 5
https: com http:///gtttuuuooooo//aaaaaaaald models
AGPL-3.0 license
SDXL
attps///gacuub..oo//Saaaaal1
Open RAIL++-M
Table 4:
Training settings for Dynami-
Table 5: Training settings for SVD 9
Crafter 63 and VideoCrafterl [2].
Config
Value
Config
Value
Optimizer
Adam
Optimizer
Adam
Learning rate
3c-5
Learning rate
lc-5
Weight decay
1c-2
Weight decay
1c-2
Optimizer momentum
81. 9209000,
Optimizer momentum
81, 32=0,, 0.999
Batch size
48
Batch size
64
Training iterations
20,000
Training iterations
20,000
Table 6: Compute resources.
Model
Iterations
GPU-type
GPU-nums
Hours
DynamiCrafter
63
20,000
A800
8
8
VideoCrafter
20,000
A800
8
8 7
SVD 9
20,000
A800
6
B.2 Training and Inference Details
We utilize the official training code of DynamiCrafter (refer to Tab. 3) to fine-tune both Dynami- Crafter [63] and VideoCrafterl [12], and reproduce the training code for SVD by ourselves. Through- the training phase, maintained consistent settings across all models, with the sole exception of incorporating our TimeNoise component for a fair comparison. Each model was fine-tuned using the WebVid-2NM dataset (2). where videos were resized and center-cropped to dimensions of 320 512 and segmented into sequences of 16 frames. In light of the discussions in Sec. 3.3 regarding the adverse effects of motion scores and the lack of precise method by SVD to compute these scores, we set a fixed motion score of 20 during training. We fix the frame rate at fps and use dynamic frame rates for DynamiCrafter
(33 and VideoCrafterl 112. Additional details can be found in Tab. and Tab. [5] Our experiments were conducted using A800-80G GPUs, and the computational costs are detailed Tab. 6 For inference, the official model codes were used for sampling (see Tab. (3). Specifically, we employed DDIM sampler with 50 steps for DynamiCrafter 63] and VideoCrafterl [2], and Heun's 2nd order method with 25 steps for SVD
B.3 Evaluation
we compute the Fréchet Video Distance (FVD) and Inception Score (IS) for 2,048 and 10,000 samples on the UCF101 dataset, re- spectively. Specifically, the FVD is calculated using the code available at battps://githu: com/SoongseiGe/TATT// with a pre-trained I3D action classification model. which can be ac- cessed at htt p=TTeeeysscccarrrr00d4sgsyyy3332 com ctm
FVD and IS. Following prior studies 26 10
(800jmmm/ii3 Seses_ttrcsscrippp The IS derived using the code from https tgan2, employing pre-trained C3D model [5I]. For this pro- cess, we sample 16 frames at fps, resize them to the default resolution of each model, and use first frame as the conditional image to generate videos. The FVD and IS are then computed between the generated videos and the ground truth videos. For DynamiCrafter [3] and VideoCrafterl [12]
7/github. wtt.:ooo/ooeeeeeeeeeeeeeeeeeeeee
which utilize text additional condition, we employ categories the textual input.
16

---

Motion Score. The Motion Scores of DynamiCrafter [63], VideoCrafterl [12] and SVD are im- plemented following SVD Specifiicllyy. we compute the motion score by resizing the video to 800 X 450, extracting dense optical flow maps using RAFT (0) between adjacent frames, calculat- ing the magnitude the 2D vector for each pixel, averaging these magnitudes spatially, and then summing them across all frames.
As for PIA [70] we calculate the motion score by making slight modification the affinity score proposed in 70), namely: motion score = 1- affinity score.
More specifically, for each video, calculate the L1 distance between each frame of and the condition frame v' in HSV space, which is denoted as d'. Then we apply this operation to all frames of video clips in the dataset and find the maximum distance value dmaz. We normalize the distance to [0, 1] via dmaas- Finally, the motion score for each frame can be calculated by m d' /dmax (Mimar mmin), where mmar and mmiin hyperparameters respectively.
0.2
User study. We ask users to compare ours with the baselines and determine which ones exhibit more dynamic and natural motion, greater temporal consistency, better alignment with the conditional image, and overall preference.
Table 7: Evaluating the combined inference (Inference.) and training (Train.) strategies. Rat. denotes a user study that ranks methods. Refer to Sec. 4.2) for detailed analysis.
VideoCrafterl
12
DynamiCrafter
63
SVD
9
Inference.
Train.
FVD4
ISt
Rat.
FVD4
IS1
Rat.
FVD4
1St
Rat.
x
x
460.3
23.98
4
382.5
21.12
4
311.0
22.03
4
/
x
450.1
24.50
1.9
342.9
22.71
2.9
277.1
22.18
2.9
X
/
452.2
24.62
2.7
334.9
21.42
1.4
272.2
23.01
1.6
/
J
443.7
25.11
1.4
332.1
22.84
1.7
272.4
25.18
1.5
C
The Influence of Adjusting the Noise Schedule
In this section, first examine commonly used strategies that involve adjusting the noise schedule to higher noise levels to bridge the
gap 14] 27 32), which, unfortunately, may exacerbate conditional image leakage. Following the SVD 9 increase noise by adjusting the Pmean in the EDM framework (28). Surprisingly, as shown in Fig. 10 we observe that larger Pmeaa values correspond reduced motion in the generated videos. We hypothesize that this is due the increased noise added to X, by larger Pmnonn making it more challenging to predict clean frames and thus more prone conditional image leakage. Consequently, the synthesized videos tend to lack dynamic and vivid motion.
D Ablation Studies for Adding Noise Method
In this section, we show another choice add noise to the conditional image. Specifically, the noisy conditional image y, at time t is obtained by
ys (1 B,) + B,e,
(20)
where B2 ~ (8.)). ..c. e ~ (). and 8.. = 1. As shown in Fig. 12 this choice leads to video discoloration SVD 19.
E
More Related Work on Diffusion Models for Image and Video Generation
Diffusion Models for Image Generation. Recently, diffusion models have achieved significant
breakthroughs in image, video, and 3D generation 24 16
37 58
59
72 For image
generation, the latent diffusion model [42 addresses computational costs by leveraging VQ-VAE (53 transform pixel-space images into compact latent representations, subsequently training diffusion models within this latent space. Building upon the latent space concept, subsequent works such as
17

---

Conditional Image
Baseline
Constant
CDM
Ours
Figure 9: The qualitative comparison between our TimeNoise and baselines mentioned in Sec. 3.2. The constant results in poor image alignment, while the CDM 25] shows low motion. Ours achieves the best visual quality.
Conditional Image
Pmean
1.2
P Pmean 0.4
Figure 10: Adjusting the noise schedule towards more noise further exacerbate conditional image leakage.
Conditional Image
Figure 11: An inappropriate motion score leads to poor temporal consistency.
18

---

Conditional Image
Figure 12: Other choice of adding noise conditional image. Adding noise via Eq. 20 causes video discoloration in SVD
Table 8: Effects of tuning M and Analytic-Init.
Model
M
FVDJ
IS
T
363.8
16.39
0.98T
345.1
16.57
0.96T
343.1
17.54
DynamiCrafter 63
0.94T
325.2
19.12
0.92T
365.8
20.49
0.9T
442.2
22.81
0.98T
324.3
16.44
0.96T
316.3
17.66
DynamiCrafter Analytic-Init
0.94T
319.6
19.13
0.92T
347.2
20.58
0.9T
378.6
22.40
Table Quantitative results of the inference strategy across varying initial time M on the UCF101 dataset.
Model
M
FVD1
ISt
DynamiCrafter (63
363.8
16.39
0.96T
316.3
17.66
0.92T
347.2
20.58
AnaalyticInii
0.88T
407.2
23.32
0.84T
535.1
24.27
0.8T
696.1
23.67
VideoCrafterl
12
353.9
18.75
0.96T
341.6
19.86
0.92T
344.3
21.58
Analytic-Ini
0.88T
368.4
21.82
0.84T
400.8
21.90
0.8T
445.6
21.21
Model
OMM
FVD4
ISt
SVD-finetune
700
311.0
22.03
500
301.9
22.00
300
290.2
21.94
Analytic-Ini
100
277.1
22.18
70
272.5
22.27
50
295.5
21.89
19

---

Algorithm Sampling from an 12V diffusion model with Analytic-Init
Require:
the conditional image yo. the sampler for diffusion models (umpler(;>))
select the initial time step M E (0, T) calculate H. and of* in initial noise distribution according to Eq. 5
XM (X p.p 07 I)
for 1 = M,..,, 1 do
X,-1 Sampler ()
end for return X
Algorithm 2 Training I2V diffusion model with TimeNoise
Require:
the clean video Xo, the conditional image you the noise schedule 04, dz the time-dependent noise distribution pt (5s)
repeat
1 ~ ((10 77)
8, (3.))
Sampling the noise level for the conditional image
~ I00
y. yo + +B,6y
Adding noise the conditional image
E ~ .)(
X, ax OfE
0 - 0 - 1/ Vo Ee (X,, ys t) e
until converged
SDXL 39 DALLE-3 8 and SD3 18 have further enhanced the performance. By exploiting such advancements in text-to-image diffusion models, numerous methods have demonstrated promising results in text-driven controllable image generation and image editing [68 22 52 As for image editing, studies such as Prompt-to-Prompt [22] and Plug-and-Play [52] have explored attention- control mechanisms over generated content, consistently delivering impressive results. For image translation, EGSDE (71) and DiffuselT (30 propose to employ an additional energy function to guide the inference process of pre-trained diffusion model.
Diffusion Models for T2V Generation. Approaches to T2V generation can be classified into two main categories. The first one involves directly generating videos based on text 46 I0 23 12 13
54
For instance, as a pioneering endeavor, Make-A-Vides [46] utilizes a pre-trained text-to-image model along with a prior network for T2V diffusion models, obviating the necessity for paired video-text datasets. VideoLDM (10) maintains the parameters of pre-trained T2I model while fine-tuning the additionally introduced temporal layers. While many models are based on U-Net [43] architecture, more recently, transformers have emerged as foundational architecture for video generation due to their scalability [21 36, 35). The second category typically entails a two-step generation process: first generating an image based on the textual input and subsequently creating a video conditioned on the text and the generated image [3 20 For instance, Emu-video [20] initializes a factorized text-to-video model using pre-trained text-to-image model and then fine-tunes temporal modules the 12V stage.
67
56
17
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations in the Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

The paper provide the full set of assumptions and a complete proof in the main text and Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results in Sec. and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
aublic/guudees/Cdeettbii.. otw.:eewwwwiisiiiooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttp:/cccpblccgguuiieeeeee
wtweeee//ooSSSeissss.ee.e www.ssssiicggsooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details (e.g., data splits, hyperparam- eters, how they were chosen, type of optimizer, etc.) necessary to understand the results in the main text and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Reporting error bars needs too much compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
The paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments in Appendix B
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htthss//ttrp///mmrrrpp..o//ppppt httpppubtttttii..c..ieeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Sec. 5

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper do not release models with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners assets (c.g., code, data, models), used in the paper, properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
25

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: For crowdsourcing experiments and research with human subjects, the paper includes the full text of instructions given to participants and screenshots, where applicable. The paper also provides details about compensation, if any was given.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
26

---

Justification: The paper describes potential risks incurred by study participants. It speci- fies whether such risks were disclosed to the subjects. The paper also indicates whether Institutional Review Board (IRB) approvals or equivalent approvals/reviews based on the requirements of the researchers' country or institution were obtained.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "i1HndEftEK", "reviewer": "Reviewer_iEBW", "summary": "This paper presents an approach for image-to-video (I2V) generation. The authors start with a conditional image leakage problem in existing works where the conditional image significantly influences the generation process thereby impacting the dynamism of the video. The authors propose an inference time and a training time strategy to mitigate the problem. The approach is evaluated on UCF101 and the authors' ImageBench dataset. The results are impressive.", "questions": "Please address the comments in the weaknesses section.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors identified a crucial problem of conditional image leakage with I2V generation. \n\n2. The authors proposed two strategies to address conditional leakage. An inference time strategy that can be readily applied to existing approaches. Another approach is applied during the training time.\n\n3. The paper is well-organized and easy to follow. \n\n4. Authors conducted a user study for qualitative evaluation. The results are impressive.", "weaknesses": "1. The authors are encouraged to provide more details about the ImageBench dataset.\n\n2. While performing the user study, what qualities are evaluated by the users?\n\n3. How is the accuracy of the generated videos estimated corresponding to the text prompt?  Do the users rate the matching score of the generated video corresponding to the text prompt?\n\n4. The realism of the generated videos is lacking. This is common for diffusion-based generated visual content.  Have the authors considered how to improve the realism of generated videos?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer iEBW\n\nWe sincerely thank Reviewer iEBW for the recognition of our work and for providing constructive comments.\n\n## Q1: Details about the ImageBench dataset.\n\nOur ImageBench dataset is designed based on two key aspects: breadth of categories and logical complexity. For breadth, we include popular categories such as nature, humans (both single and multi-person), animals, plants, buildings, food, arts, and vehicles. For complexity, we incorporate elements like numerals, colors, and complex scenes. Following these principles, we collected 100 images from various sources and T2I models like SDXL[1] and UniDiffuser[2]. We will add this in the revised version.\n\n [1] Sdxl: Improving latent diffusion models for high-resolution image synthesis.\n\n [2] One transformer fits all distributions in multi-modal diffusion at scale.\n\n## Q2: While performing the user study, what qualities are evaluated by the users?\n\nAs noted in lines 226-228 of the original paper, users were asked to conduct pairwise comparisons between our method and the baselines, assessing dynamic motion, temporal consistency, image alignment, and overall performance.\n\n## Q3: Add the evaluation about the text alignment.\n\nWe appreciate the constructive feedback provided. Following the suggestion, we add text alignment evaluation using the CLIP-text score[1] and a user study in **Rebuttal Table B**. Since SVD does not utilize text prompts as input, we conducted these evaluations only for VideoCrafter1 and DynamiCrafter. The results demonstrate that our approach achieves comparable text alignment performance with the baselines, suggesting that our strategy does not adversely affect text alignment quality.\n\n[1]: Learning Transferable Visual Models From Natural Language Supervision\n\n**Rebuttal Table B.** Text alignment performance comparison on the ImageBench dataset. < Method >-naive-tune and < Method >-TimeNoise represent finetuned models with naive strategy and our training strategy respectively, using the same setup and dataset for a fair comparison. User preference indicates the percentage of users who preferred our method over the baselines. DC. and VC. denote DynamiCrafter and VideoCrafter1.\n\n| Model | CLIP-text Score ↑ | User Preference ↑ |\n|-------|--------------------|------------------|\n| DC. / DC. + Analytic-Init | 0.274 / **0.277** | **53%** / 47% |\n| DC.-naive-tune / + Analytic-Init | **0.269** / 0.266 | 48% / **52%** |\n| DC.-naive-tune / DC.-TimeNoise | **0.269** / 0.267 | 49% / **51%** |\n||||\n| VC. / VC. + Analytic-Init | **0.252** / 0.251 | **52%** / 48% |\n| VC.-naive-tune / + Analytic-Init | 0.247 / **0.255** | 45% / **55%** |\n| VC.-naive-tune / VC.-TimeNoise | **0.247** / 0.242 | 46% / **54%** |\n\n## Q4: Future work on improving the realism of generated videos.\n\nWe appreciate the insightful comments. We think there are two promising ways for enhancing the realism of the generated videos. One promising direction is to explore advanced model architectures, such as the Transformer architecture. By refining the architecture to better capture long-range dependencies and spatial-temporal coherence, we may enhance the realism and consistency of the generated videos. Another possible aspect is to scale up the diffusion model. Scaling the model can involve increasing the depth and width of the network, as well as using larger datasets to improve\nquality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cdk0ELnnp0", "reviewer": "Reviewer_NwFJ", "summary": "This paper investigates and proposes solutions for the problem of “conditional image leakage” in image-to-video generation. The authors claim that existing image-to-video diffusion models over-rely on the conditional image at large diffusion timesteps when the inputs are too noisy, leading to static video output. To address this issue, the authors propose an inference-time generation process that skips those large timesteps. The authors also propose adding noise to the conditional image during training to mitigate conditional image leakage.", "questions": "What are the motion magnitudes of SVD in Figure 3 and Figure 7? \n\nAdding different noises to the conditioning image at different time steps requires the conditioning image to be embedded multiple times. How does this affect inference time?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper identifies the problem of conditional image leakage and shows that existing methods suffer from static motion at large diffusion timesteps. The two proposed solutions empirically boost FVD and IS on UCF101 and bring significant improvements to motion magnitude while maintaining a similar image alignment and temporal consistency in the user study. These show that the proposed approach is indeed useful in generating more dynamic videos.", "weaknesses": "There are a lot of prior works [a, b, c] that deal with noise initialization in video diffusion models that are highly related to the problem of conditional image leakage and the proposed inference technique. There are, however, no discussions on the differences between these techniques. Notably, FreeInit [a] and FrameInit [c] perverse the low-frequency component of the initial image – which is at odds with the proposed theory. I believe an in-depth discussion and a comparison with these approaches would help to establish the thesis of this paper. Additionally, PIA [d] also allows motion control – comparing the proposed motion-adding techniques with PIA’s motion control would also be helpful.\n\n[a] FreeInit: Bridging Initialization Gap in Video Diffusion Models\n\n[b] Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models\n\n[c] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation\n\n[d] PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models\n\nSecondly, I am not entirely convinced by the authors’ explanation of conditional image leakage. The authors claim that models learn a shortcut by copying the initial image at large diffusion timesteps, leading to static video output. However, models like VideoCrafter and SVD do produce outputs of diverse motion of some degree, as shown in their papers. What are the authors’ explanations for these motions? Where are they coming from? Trying to copy the conditional image when the input is otherwise noise is likely the optimal solution and would be learned – but that is not a sufficient condition for static output.\n\nThirdly, how are evaluations normalized for motion magnitude? Methods like SVD and PIA have motion control. Since the evaluation is focused on “dynamic motion”, perhaps choosing a different motion parameter for these methods would lead to different results in the user study. How are those parameters chosen? Do baseline SVD, SVD after finetuning, and SVD with the inference trick interpret the same motion scale parameter in the same way? One way to validate this is to plot the input motion scale parameter against the output motion magnitude (e.g., optical flow) for parameter selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer NwFJ\n\nWe thank Reviewer NwFJ for the valuable comments. \n\n## Q1: Add comparisons.\n\n### Q1-1: Add comparison with related noise initialization methods.\n\n(1) As suggested, we add comparisons with FreeInit[a], Progressive Noise[b] and FrameInit[c]. Our Analytic-Init outperforms all baselines across all metrics, as demonstrated in **Rebuttal Table C**. Specifically:\n-  compared with FrameInit: we achieve much higher motion scores. This is because FrameInit duplicates the conditional image into a static video and uses it as coarse layout guidance,which can limit motion degree.\n- compared with FreeInit: we reduce inference time by more than half, while achieving a slight enhancement in performance. This is because FreeInit's iterative refinement process, involving the sampling of a clean video, adding noise, and regenerating the video, increases its overall inference time.\n- compared with Progressive Noise: we achieve better performance. This is because our inference framework generates videos from an earlier time step, while the baseline can not handle this training-inference gap.\n\n(2) Additionally, we clarify that our initialization method enhances temporal consistency (referred to as 'low frequency' in related work), consistent with findings in prior work ( see 5th vs 9th columns in Figure 4 of the original paper). Our high-motion capability (referred to as 'high frequency' in related work) is improved by starting the generation process at an earlier time step rather than initialization, avoiding the unreliable late-time steps of I2V-DMs.\n\nThank you for highlighting these important related works. We will incorporate the above discussions in the final version.\n\n**Rebuttal Table C.** Comparison of different noise initialization methods. Output MS is the motion score of the generated videos, computed via an optical flow network. User Rate. indicates the average rank assigned by users across all methods from overall aspects. Inference Time refers to the duration required to generate a video.\n| Method            | FVD ↓   | IS ↑   | Output MS ↑ | User Rate ↑ | Inference Time ↓ |\n|-------------------|---------|--------|-------------|-------------|------------------|\n| FrameInit         | 380.7   | 20.09  | 32.16       | 4.57        | 24.3s/it         |\n| FreeInit          | 347.4   | 22.66  | 46.24       | 1.95        | 49.6s/it         |\n| Progressive Noise | 358.1   | 21.35  | 49.76       | 3.31        | 23.3s/it         |\n| Analytic-Init     | **342.9** | **22.71** | **50.08**   | **1.77**   | **22.6s/it**     |\n\n\n\n### Q1-2: Add comparison with PIA.\n\nAs suggested, we apply our inference strategy to PIA. Our method consistently enhances the motion degree, regardless of the initial motion level in PIA, as demonstrated in **Rebuttal Table D in the reponse PDF**. This improvement suggests the presence of conditional image leakage in PIA. It is important to note that our method is orthogonal to approaches that introduce additional signals, such as PIA and SVD, and enables them control motion more precisely (see Common Concern 1).\n\n## Q2: The authors claim models learn a shortcut by copying the image, yet they still produce outputs with motion.\n\nWe clarify that the static video example in Figure 2 is a schematic representation. We aim to highlight that these models tend to generate videos with **reduced motion** due to over-reliance on the conditional image at large diffusion timesteps (see lines 6, 34, 103 in the original paper). In some cases, this over-reliance might lead to nearly static videos, as exemplified in Figure 2. We repeat the experiments three times using different random seeds and conducte a statistical analysis using a two-sample t-test between the baselines and ours. As demonstrated in **Rebuttal Table E in the response PDF**, our method achieved higher motion scores with a significance level of p < 0.05 compared to all baselines. We will make it clear in the final version.\n\n## Q3: The principle for motion score chosen. Plot the input motion score against the output motion score.\n\nFor the input motion score chosen, we select a range of motion scores between 5 and 200 on the ImageBench. A score of 20 achieves high visual quality, including high temporal consistency and natural motion, and is used as a baseline. In the remaining experiments (Table 1, Figure 7, Figure 9), we directly use this 20 motion score for the baseline SVD, SVD-tune, and SVD with Analytic-Init. As suggested, we report the input motion scale against the output motion scale in **Rebuttal Table A** and **Figure 1 in the response PDF**. The results show that our method enables SVD control motion degrees more precisely by consistently achieving a lower error between the motion score of outputs and input motion score. See more detailed analysis in Common Concern 1.\n\n## Q4: The motion score in Figure 3 and Figure 7\n\nIn Figure 3, we use the default motion score of 127 in SVD, which tends to show camera movement with poor temporal consistency. In Figure 7, as detailed in Q3, we use a motion score of 20 for its high visual quality. As discussed in Common Concern 1, our method consistently achieves a lower motion score error, regardless of the input motion score in SVD.\n\n## Q5: How does TimeNoise affect inference time?\n\nDuring inference, we use the conditional image with a fixed noise level across all time steps, similar to CDM[1], which does not affect inference time. This is because, during training, the model learns to handle conditional images with varying noise levels. In our initial experiments, we tested noise levels at 0, 5, and 10, finding that levels 0 and 5 exhibit similar performance, while a level of 10 sacrifices image alignment and temporal consistency. Based on these findings, we directly use the clean conditional image across all time steps for simplicity. We will include these details in the final version.\n\n[1] Cascaded diffusion models for high-fidelity image generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rdn9c7MRny", "reviewer": "Reviewer_hjyn", "summary": "The paper points out that existing image-to-video diffusion models can lead to videos without significant/desired amount of motion. Some evidence is presented to show that this is due to what the authors call \"conditional image leakage\" where the model places too much emphasis on the conditional image and all the generated frames look too similar to the conditional image. Two strategies are proposed to overcome this issue. The inference strategy starts the reverse diffusion process at an earlier time and uses improved initial noise distribution parameters using a closed-form expression to match the noise parameters at training time. The training strategy develops a time-dependent noise distribution to have higher noise at larger t to place lesser emphasis on the conditioning image for larger t values. Experiments indicate improved video generation using either or both of these strategies.", "questions": "No additional questions.", "limitations": "Yes, authors have addressed some limitations of their method.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The methods presented in the paper are well-motivated and make intuitive sense (although there appear to be some technical issues as noted in the weaknesses below).\n\n2. Results: The quantitative results show clear improvements over baseline. The video examples submitted also show clear improvements over previous methods\n\n3. Relationship to existing works: the authors have done a good job presenting related works carefully and how existing methods may also inherently overcome some issues, if not all.", "weaknesses": "1. Issues with derivation: For deriving the expression of better noise initialization, the authors present their proof in Appendix A. The second and third terms of (7) are considered constants for the purpose of optimizing the noise distribution. I don't follow this and think there may be a technical error here. It is written that since $q_M(X_M)$ and $p_M(X_M)$ are independent, the entropy $H(\\mathcal{N}(X_M; \\mathbf{\\mu}_q, \\Sigma_q))$ and $H(q_M(X_M))$ can be considered constants for optimizing the parameters $\\mathbf{\\mu}_q$ and $\\Sigma_q$. I don't think this is correct as these parameters influence both $H(\\mathcal{N}(X_M; \\mathbf{\\mu}_q, \\Sigma_q))$ and $H(q_M(X_M))$. Can the authors please explain this in detail?\n\n2. Quantifying motion: The authors say that with previous works the models could in principle just output the conditioning image several times and still lead to lower error or improved quality metrics like FID and IS. However, they still use the same metrics for their experiments. Perhaps quantifying amount of motion or even better, quantifying realism of motion directly is important. \n\n3. Figure 9 is not clear. What are orange and blue colors indicating. Please explain this figure in detail and how it shows that the proposed method is considered better by the user study. I see very little differences going from <method> to <method>-tune in the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer hjyn\n\nWe sincerely thank Reviewer hjyn for the constructive and valuable comments. The concerns are addressed as follows.\n\n## Q1: Issues with derivation.\n\nWe clarify that our objective is to optimize $\\mu_p$ and $\\sigma_p^2$, the parameters of $p_M(X_M)=N(X_M; \\mu_p, \\sigma_p^2 I)$, rather than $\\mu_q$ and $\\Sigma_q$ (as stated in Proposition 1, line 155 of the original paper). Note that $\\mu_q$ and $\\Sigma_q$ represent the expectation and covariance matrix of the forward marginal distribution $q_M(X_M)$, which is determined by the forward diffusion process and is independent of the $\\mu_p$ and $\\sigma_p^2$ in $p_M(X_M)$. Given this independence, the second and third terms in Equation (7) can be disregarded for the purpose of optimization. Therefore, the optimization objective can be formulated as:\n$$\n    \\min_{\\mu_p,\\sigma_p^2} D_{KL}(q_M(X_M)||p_M(X_M)) \\Leftrightarrow \\min_{\\mu_p,\\sigma_p^2} D_{KL}(N(X_M; \\mu_q, \\Sigma_q)||N(X_M; \\mu_p, \\sigma_p^2 I))\n$$\nWe will improve the writing of Appendix A in the final version.\n\n## Q2: The authors say that with previous works the models could in principle just output the conditioning image several times and still lead to lower error or improved quality metrics like FID and IS. However, they still use the same metrics for their experiments. Perhaps quantifying amount of motion or even better, quantifying realism of motion directly is important.\n\nWe clarify that models that repeatedly output the conditioning image might achieve a lower **training loss**, as defined by Eq. (2) in the original paper, rather than a lower evaluation error or improved video quality metrics like FVD and IS. In fact, as demonstrated in Table 1 of the original paper, such repetitive behavior can lead to poorer FVD and IS, since these metrics can evaluate the dynamic qualities of videos and static conditioning images lack the inherent dynamics. It's important to note that FVD and IS are video evaluation metrics based on an action classification model rather than image-based metrics like FID and IS, which can reflect video dynamics and video quality.\n\nAs suggested, we also incorporate the motion score metric to quantify the amount of motion, which is computed by obtaining dense optical flow maps with RAFT [1] between adjacent frames, calculating the magnitude of the 2D vector for each pixel, and summing these magnitudes across the frame. As demonstrated in **Rebuttal Table F**, our approach yields substantially higher motion scores compared to all baseline methods. We will add this in the final version. Quantifying the realism of motion is indeed crucial, yet, to the best of our knowledge, there currently exists no specific automatic quantitative metric to evaluate this aspect directly. Our user study, which covers overall aspects, takes this into account. We welcome any suggestions from the reviewers regarding additional metrics for evaluating motion realism.\n\n[1] Raft: Recurrent all-pairs field transforms for optical flow.\n\n**Rebuttal Table F**. Motion score results on the ImageBench dataset. < Method >-naive-tune represents a naively fine-tuned baseline using the same training setup and dataset as our TimeNoise training strategy, ensuring a fair comparison.\n\n| Model | Motion Score ↑ |\n|-------|-----------------|\n| DymiCrafter | 50.96 |\n| + Analytic-Init | **71.04** |\n|||\n| DymiCrafter-naive-tune | 31.68 |\n| + Analytic-Init | **50.08** |\n| + TimeNoise | **72.32** |\n|||\n| VideoCrafter1 | 63.36 |\n| + Analytic-Init | **139.04** |\n|||\n| VideoCrafter1-naive-tune | 62.72 |\n| + Analytic-Init | **65.12** |\n| + TimeNoise | **64.80** |\n|||\n| SVD | 16.64 |\n| + Analytic-Init | **19.68** |\n|||\n| SVD-naive-tune | 9.60 |\n| + Analytic-Init | **20.64** |\n| + TimeNoise | **20.96** |\n\n## Q3: Explanation of Figure 9.\n\nFigure 9 shows user preference percentages for our methods versus baselines. The left side displays baselines, while the right side shows our inference (Analytic-Init) and training (TimeNoise) strategies. Orange indicates the preference for our strategies and blue for the baselines. For instance, in the first row at the top left, 84.2% preferred our inference strategy over the original DynamiCrafter's output (15.8%) from an overall perspective. The < method > and < method >-tune denote the original model and naively finetuned versions, serving as baselines. The < method >-tune uses the same training settings and dataset as our TimeNoise strategy, ensuring a fair comparison. This is crucial because I2V-DMs often use private datasets with varying data filtering, resolutions, and unknown training settings. Controlling these variables ensures a fair comparison.\n\nIn summary, we validate our inference strategy against the original method and a naively finetuned version, and our training strategy against a naively finetuned version. As discussed in the original paper (lines 238-245), our strategies significantly enhance video dynamism and natural motion, maintaining image alignment and temporal consistency, thus achieving superior results overall. We will rename < method >-tune as < method >-naive-tune for clarity and improve the caption of Figure 9 in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqyDoxhQ1L", "reviewer": "Reviewer_f9AW", "summary": "This paper argue that I2V-DMs tend to overly rely on the conditional image at large time steps, resulting in videos with less dynamic motion.\nTo address this, they propose using the KL divergence between the initial noise distribution and the actual marginal distribution to enhance the amplitude of motion while preserving video quality.\nBesides，the paper propose a time-dependent noise distribution for the conditional image during training process.", "questions": "Can the authors provide more insights into the selection of hyperparameters (Line 230 Implementation Details) for the time-dependent noise distribution?", "limitations": "yes, the authors have discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Motivation. The challenge of conditional image leakage presented in the paper is a credible issue and represents a significant academic problem.\n\nThe paper is exceptionally well-organized, making complex concepts easily digestible for the reader.", "weaknesses": "1. Claim. Regarding the conditional image and the diversity of content changes (referred to as 'motion' in this paper), it is, in fact, a matter of trade-off. The essence of image to video generation is to leverage the conditional image to enhance video quality. Therefore, the Proposition 1 and Training Strategy proposed by the authors are essentially fine-tuning techniques that favor the diversity of motion.\n\n2. Additionally, the foundation model of video generation guided only by the first frame inherently contains uncertainty in its output; it does not know whether the user wants to generate a video with significant or minimal motion. Some recent studies (last frame guided[1], motion control[2])have demonstrated that incorporating additional guidance signals can indeed achieve big and precise motion. \n\nParticularly, the works DragNUWA and MotionCtrl have shown that for the image-to-video generation task, it is still possible to generate significant motion. Therefore, the foundation model of SVD has the potential to directly generate substantial motion. In this context, the initial version of SVD, which could only generate minor motion, might be due to the lack of a specific condition or guidance signal for significant motion, rather than an inherent incapability. From a subjective standpoint, not all applications require substantial motion.\n\n[1] ToonCrafter: Generative Cartoon Interpolation\n[2] MotionCtrl: A Unified and Flexible Motion Controller for Video Generation\n[3] DragAnything: Motion Control for Anything using Entity Representation\n[4] DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory \n\n3. Experiments. The experiments appear to be somewhat confusing and not entirely convincing. \nRegarding the variables of the related training strategy displayed in Figure 8, the reviewer has not found the corresponding ablation study, which is a core contribution of this paper. Additionally, the author has not provided an ablation study on the 'M' of Proposition 1, such as testing performance changes between 0.6T and 1T, which is central to the first contribution. If the reviewer has overlooked it, please point it out.\n\nAdditionally, Figure 9 also appears to be somewhat confusing. Was a user study not conducted on the Analytic-Init? What does '-tune' signify—does it mean finetuning (training strategy)? If so, why does it appear in the inference phase?\n\n--------------------\nAfter reading the author’s and other reviewers' feedback, I have adjusted my score to 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer f9AW\n\nWe thank reviewer f9AW for the valuable and constructive comments. We address the concerns as follows.\n\n## Q1: Claim. The conditional image and the diversity of content changes are, in fact, a matter of trade-off. The essence of image-to-video generation is to leverage the conditional image to enhance video quality. Proposition 1 and the Training Strategy proposed by the authors are essentially fine-tuning techniques that favor the diversity of motion.\n\nWe appreciate the reviewer for the comment but disagree with our highest respect. While the conditional image for video quality and motion diversity can be a trade-off, **we emphasize that our results simultaneously achieve dynamic motion and high video quality, without sacrificing either aspect**. As demonstrated in Table 1, we achieve superior FVD and IS metrics compared to all baselines, indicating higher video quality. The user study in Figure 9 further confirms that our method outperforms all baselines in overall aspects, including video quality and dynamic motion. The results in **Rebuttal Table A** show that our method achieves a lower motion score error than all baselines, underscoring that our method helps to control motion degrees more precisely. \n\n## Q2. Recent studies show incorporating additional guidance signals can achieve significant motion and control the motion degree. From a subjective standpoint, not all applications require substantial motion.\n\nWe clarify that our method is orthogonal to the approaches that introduce additional signals and our method enables them control motion degrees more precisely. We validate this by consistently achieving a lower error between the motion score of the generated video and the given desired motion score, regardless of the input motion score in SVD (see **Rebuttal Table A**). More details are provided in Common Concern 1. DragAnything, DragNUWA, MotionCtrl, and ToonCrafter are novel controllable methods that achieve great performance on controllable video generation. We will include a discussion with these methods in the Related Work section in the final version\n\n## Q3: Ablation studies.\n\nWe did a systematic analysis of each hyperparameter in our proposed strategy in the original paper. \n\nFor the start time $M$ in Analytic-Init, the qualitative and quantitative results are reported in Figure 4, Table 7 and Table 8. As discussed in lines 144-146 and lines 159-162 of the original paper, an appropriate $M$ can enhance performance by increasing motion without compromising other performance. A too-small $M$ delivers poor visual quality due to the training-inference gap. Analytic-Init helps to mitigate it, especially with a smaller $M$.\n\nFor the two hyperparameters in the training strategy, the qualitative results and corresponding analysis are shown in Figure 5 and described in lines 183-195 in the original paper. Specifically, higher noise levels, such as those produced by a concave function ($a$ < 1), enhance dynamic motion but reduce temporal consistency and image alignment. For the maximum noise level $\\beta_m$, a higher $\\beta_m$ enhances dynamic motion but decreases temporal consistency and image alignment, while a lower $\\beta_m$ leads to less motion. \n\n## Q4: In Figure 9, was a user study not conducted on the Analytic-Init? What does '-tune' signify—does it mean finetuning (training strategy)? If so, why does it appear in the inference phase?\n\nThe inference strategy in Figure 9 is denoted as Analytic-Init. The < method >-tune represents a naively finetuned baseline using the same training setup and dataset as our TimeNoise training strategy, ensuring a fair comparison. We validate our inference strategy on both the original baseline and a naively finetuned baseline. The < method >-tune baseline is crucial because I2V-DMs often use private datasets with varying data filtering, resolutions, and unknown training settings. Controlling these variables ensures a fair comparison. We will rename < method >-tune as < method >-naive-tune and label the inference strategy as Analytic-Init for clarity in the final version.\n\n## Q5: Insights into the selection of hyperparameters\n\nAs discussed in lines 184-185 of the original paper, firstly, the principle of $\\mu(t)$ is to increase monotonically with time step. This ensures that higher noise levels are favored at later time steps, which carries a higher risk of leakage. To formalize this, we define $\\mu(t)$ as a power function: $\\mu(t)=2t^a-1, a>0$. This choice allows us to flexibly tune $a$ to achieve various monotonic behaviors, where smaller values of $a$ indicate that more of the later time steps will sample higher noise levels. Specifically:\n\n- $a=1$ corresponds to a linear increase,\n- $a<1$  indicates a concave function, suggesting a faster increase in noise levels over time, and\n- $a>1$ represents a convex function, indicating a slower increase in noise levels over time.\n\nFor the choice of the maximum noise level $\\beta_m$, the only principle is that it should be greater than 0. We refer to popular choices of maximum noise levels in VE-SDE[1,2] (e.g. 80-100) and experiment with parameters around these values.\n\n[1] Score-based generative modeling through stochastic differential equations.\n\n[2] Elucidating the Design Space of Diffusion-Based Generative Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Common Concerns from reviewers

## Common Concern 1 (from   NwFJ and f9AW): Incorporating additional guidance signals can achieve significant motion and control the motion degree. Show the input motion score against the output motion magnitude.

We clarify that **our method is orthogonal to the approaches that introduce additional signals and enables them to the control motion degree more precisely**. As demonstrated in **Rebuttal Table A** and **Figure 1 in the response PDF**, our method consistently achieves a lower error between the motion score of the generated video and the given desired motion score, regardless of the input motion score. Notably, the baseline tends to generate videos with a significantly lower motion score than the input motion score, suggesting the presence of conditional image leakage. Our method effectively reduces this discrepancy.

In addition, we emphasize that our method not only enables more precise control of the motion degree but also **enhances its naturalness**. As discussed in the original paper (lines 123-130) and illustrated with more examples in **Figure 2 of the response PDF**, the baseline often produces static objects with pronounced camera movements to meet high motion scores. In contrast, our approach generates videos with natural and vivid object movements. This suggests that a simple scalar condition may not be sufficient to address the fundamental challenge facing I2V-DMs, which is to generate clean videos primarily from noisy inputs. Our method offers a more effective solution, enhancing realism and fluidity in the generated videos.



**Rebuttal Table A.** Results on the ImageBench dataset. SVD-naive-tune and SVD-TimeNoise are finetuned SVD with naive finetuning and our TimeNoise respectively, using the same setup for a fair comparison. Input MS is SVD's input motion score. Output MS is the average motion score of generated videos. Error is the absolute difference between them. User Preference is percentages favoring our method. Note that the motion score is implemented based on the description provided in the SVD paper, as the actual code is unavailable. For the first two experiments (SVD-naive-tune and SVD-TimeNoise), both the input and output MS are consistently calculated, ensuring that the error measure is reliable. In the case of the original SVD (the third experiment), the error may not be as robust, but it is included for reference.

| Input MS | Output MS $\uparrow$ | Error $\downarrow$ | User Preference $\uparrow$ |
|----------|-------------|---------|-------------------|
| | **SVD-naive-tune / + Analytic-Init**|
| 5     | 2.08 / **4.64** | 2.92 / **0.36** | 32.0% / **68.0%** |
| 10    | 5.12 / **8.16** | 4.88 / **1.84** | 29.5% / **70.5%** |
| 20    | 9.60 / **20.64** | 10.4 / **0.64** | 16.5% / **83.5%** |
| 40    | 19.84 / **34.08** | 20.16 / **5.92** | 21.0% / **79.0%** |
| 80    | 52.96 / **65.12** | 27.04 / **14.88** | 32.5% / **67.5%** |
| 100   | 55.84 / **83.68** | 44.16 / **16.32** | 19.5% / **80.5%** |
| 127   | 55.52 / **111.20** | 71.48 / **15.80** | 34.0% / **66.0%** |
| 200   | 64.16 / **133.92** | 135.84 / **66.08** | 27.5% / **72.5%** |
| | **SVD-naive-tune / SVD-TimeNoise** |
| 5     | 2.08 / **6.72** | 2.92 / **1.72** | 19.5% / **80.5%** |
| 10    | 5.12 / **9.44** | 4.88 / **0.56** | 9.0% / **91.0%** |
| 20    | 9.60 / **20.96** | 10.4 / **0.96** | 7.2% / **92.8%** |
| 40    | 19.84 / **44.80** | 20.16 / **4.80** | 11.5% / **88.5%** |
| 80    | 52.96 / **80.48** | 27.04 / **0.48** | 15.5% / **84.5%** |
| 100   | 55.84 / **97.12** | 44.16 / **2.88** | 18.5% / **81.5%** |
| 127   | 55.52 / **113.76** | 71.48 / **13.24** | 23.0% / **77.0%** |
| 200   | 64.16 / **150.24** | 135.84 / **49.76** | 35.5% / **64.5%** |
| |**SVD / +Analytic-Init** |
| 5     | 4.32 / **5.12** | 0.68 / **0.12** | 23.0% / **77.0%** |
| 10    | 8.96 / **9.92** | 1.04 / **0.08** | 31.5% / **68.5%** |
| 20    | 16.64 / **19.68** | 3.36 / **0.32** | 16.5% / **83.5%** |
| 40    | 31.04 / **40.48** | 8.96 / **0.48** | 21.0% / **79.0%** |
| 80    | 72.16 / **78.72** | 7.84 / **1.28** | 16.0% / **84.0%** |
| 100   | 92.96 / **104.64** | 7.04 / **4.64** | 38.5% / **61.5%** |
| 127   | 112.96 / **125.28** | 14.04 / **1.72** | 42.0% / **58.0%** |
| 200   | 190.40 / **206.40** | 9.60 / **6.40** | 42.5% / **57.5%** |

Meta-review: This paper discusses the issue of conditional image leakage in an image-to-video generation model where the conditioning image may potentially bring unintended control into the generation of video dynamics. The paper proposes two ideas to mitigate the issue, leading to strong empirical performances. 

The paper was reviewed by four experts, who questioned various aspects of the paper. Authors provided a strong rebuttal, addressing the concerns, resulting in all reviewers inclining towards acceptance. AC agrees with the reviewers that the paper considers an important problem in conditional video synthesis, and the proposed approach shows solid results. Thus, AC recommends acceptance. Authors should revise the paper as per the reviewers' suggestions as well as incorporate all the comparisons that were provided in the rebuttal in the camera-ready.
Decision: Accept (poster)