IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis
Authors: Shitong Shao
zikai zhou
Bai LiChen
Haoyi Xiong
Zeke Xie
Venue: ICLR 2025 Poster (2025)
Keywords: IV-mixed Sampler
Video Synthesis
Inference-heavy Algorithms
Training-free
Primary area: generative models
TL;DR: We design a novel training-free algorithm IV-mixed Sampler that leverages the strengths of image diffusion models to assist video diffusion models surpass their current capabilities.

Abstract: Exploring suitable solutions to improve performance by increasing the computational cost of inference in visual diffusion models is a highly promising direction. Sufficient prior studies have demonstrated that correctly scaling up computation in the sampling process can successfully lead to improved generation quality, enhanced image editing, and compositional generalization. While there have been rapid advancements in developing inference-heavy algorithms for improved image generation, relatively little work has explored inference scaling laws in video diffusion models (VDMs). Furthermore, existing research shows only minimal performance gains that are perceptible to the naked eye. To address this, we design a novel training-free algorithm IV-Mixed Sampler that leverages the strengths of image diffusion models (IDMs) to assist VDMs surpass their current capabilities. The core of IV-Mixed Sampler is to use IDMs to significantly enhance the quality of each video frame and VDMs ensure the temporal coherence of the video during the sampling process. Our experiments have demonstrated that IV-Mixed Sampler achieves state-of-the-art performance on 4 benchmarks including UCF-101-FVD, MSR-VTT-FVD, Chronomagic-Bench-150/1649, and VBench. For example, the open-source Animatediff with IV-Mixed Sampler reduces the UMT-FVD score from 275.2 to 228.6, closing to 223.1 from the closed-source Pika-2.0.

Published as a conference paper at ICLR 2025
IV-Mixed Sampler: LEVERAGING IMAGE DIFFUSION MODELS FOR ENHANCED VIDEO SYNTHESIS
Shitong Shao 10
Zikai Zhou' 1.0
Lichen Bai'
Haoyi Xiong²
Zeke Xie
'Hnng Kong University of Science and Technology (Guangzhou) Baadd Inc. {sshac213, kkkizhou, lichenbai, jekexiej@hkusttgzeeeee en hacyi. xicegg freicco. org, :Equal Contribution, #:Corresponding author
ABSTRACT
Exploring suitable solutions improve performance by increasing the computa- tional cost of inference in visual diffusion models is a highly promising direction. Sufficient prior studies have demonstrated that correctly scaling up computation in the sampling process can successfully lead to improved generation quality, en- hanced image editing, and compositional generalization. While there have been rapid advancements in developing inference-heavy algorithms for improved im- age generation, relatively little work has explored inference scaling laws video diffusion models (VDMs). Furthermore, existing research shows only minimal performance gains that are perceptible the naked eye. To address this, we design a novel training-free free algorithm /V-Mixed Sampler that leverages the strengths of image diffusion models (IDMs) to assist VDMs surpass their current capabilities. The core of IV-Mixed Sampler is to use IDMs to significantly enhance the quality of each video frame and VDMs ensure the temporal coherence of the video during the sampling process. Our experiments have demonstrated that /V-Mixed Sampler achieves state-of- satte-of-the-ann performance on benchmarks including UCF-101- 101-FDD, MSR-VTT-FVD. Chronomagic- Beeee-Benhhhhi Beench-15011644 550/ and VBench. For example, the open-source Animatediff with IV-Mixed Sampler reduces the UMT-FVD score from 275.2 228.6, closing to 223.1 from the closed-source Pika-2.0. Our code is released at https://github. wwtp:/tthhu..c.iieeeeeeeee wwwpooieoaoooooooeeeeeeeeeaaeeee laa--aalV-mmieed mixidd-Sampler..... Our project page can be found in https://klayand haw.lllayynnngggu...oooo wwt.uuu..ioommiixxxeemem
INTRODUCTION
In the large foundation models era, maximizing the inference potential of foundation mod- els (Blattmann et al., 2023; Touvron al., 2023) that require high pre-training costs has become a research staple for academics (Wu et al., 2023; Wei et al., 2022). Efficient plug-and-pe rithms (Zhang et al., 2023; Mou et al., 2024) can significantly drive large-scale models to reach their full potential and outperform the original counterparts due to low trial-and-error costs. In contrast to popular inference-heav heavy algorithms (Wei et al., 2022; Chen et al., 2023b; Chefer et al., 2023; Chen et al., 2024b), which consistently emerge in the large language model (LLM) and image diffuuson model (IDM) field, text-to-video (T2V) synthesis still faces the prevalent challenge of low-quality synthesized videos that lack semantic faithfulness (Guo et al., 2023; Wang al., 2023). This limita- tion severely hampers the deployment and application of video diffusion models (VDMs). Motivated by the success of inference scaling laws in both LLMs and IDMs, we inevitably wonder if it is pos- sible to design an inference-heavy algorithm that is effective and plug-and-play on VDMs, allowing us enhance VDMs' inference performance? This answer is obvious as VDM and IDM are the- oretically identical (Song et al., 2023c). However, designing unique and outstanding training-fres algorithms based on the properties of VDM remains a significant challenge.
algo-
Motivation. VDM-based training- free algorithms typically face significant challenge: their per- formance ceiling is constrained by the VDM itself. To be specific, the videos synthesized from most open-source VDMs (Guo et al., 2023; Wang al., 2023) exhibit several inherent problems, such as weak semantic consistency between the videos and the prompts, as well as low quality. A widely accepted perspective (Wu al., 2023; Zhai et al., 2024) on this phenomenon is that underper-

---

Published as a conference paper at ICLR 2025
Qualit
Visa Quality
Figure Visualization of TVmixedd Sampler and the standard DDIM sampling on Animatediff and VideoCrafter Unlike prior heavy-inferenne approaches (Guo et al., 2024; Wu et al., 2023), IV-mixed Sam- pler is able to significantly improve the fidelity of the video while guaranteeing semantic faithfulness.
forming VDMs cannot overcome their intrinsic limitations because they are trained on low-quality, small-scale T2V datasets. In contrast, IDMs can now reliably serve commercial application scenar- ios (Betker et al., 2023; Liu et al., 2024a), thanks to the well-established dataset ecosystem created by the AIGC community. This discrepancy naturally prompts us to consider whether we can view IDMs as tools to enhance VDMs and how to make IDMs effective assistants for VDMs.
Since diffusion models employ multi-step sampling mechanism and the quality of synthesized samples from IDMs is significantly higher than that from VDMs, it is reasonable to use IDMs and VDMs alternately for score function estimation throughout the reverse sampling process. However, this straightforward paradigm leads to loss of temporal coherence the synthesized videos during initial empirical explorations. Inspired by SDEdit (Meng et al., 2021), enhance the quality of each frame at every denoising step by performing the following additional operation: 1) first adding Gaussian noise and 2) then denoising using IDMs. We found that this form of inference significantly impairs performance. Through empirical exploration, we identified the issue as the "first adding Gaussian noise" step, which introduces excessive randomness. Replacing this step with deterministic modeling paradigm (i.e,, deterministic sampling) yields stable performance gains.
In this paper, we consider the widely used deterministic function HDDM--Ivverson (Mokady et al., 2023) to inject perturbations into the video. To be specific, we integrate IDM and VDM using the base operator: first performing DDIM-Inversion, followed by applying operations similar to DDIM before each denoising step. Additionally, the observation that performing DDIM-Inversion with IDM first significantly outperforms performing DDIM-Inversion with VDM first further supports the conclusion that IDMs trained high-quality datasets can positively impact the video sampling process. Motivated by this, we further explore the upper bound the performance gain that IDM provides for video synthesis. We primarily extend the paradigm of the single-step diffusion process and the single-step reverse process to multiple steps. Then, to ensure inference efficiency, we inves- tigate all possible combinations the two-step diffusion process and the two-step inverse process, collectively naming this series of algorithms /V-mixed Sampler.
Contribution. Specifically. 1) construct IV-mixdd Sampler under rigorous mathematical framework and demonstrate, through theoretical analysis, that it can be elegantly transformed into standard inverse ordinary differential equation (ODE) process. For the sake of intuition,
2

---

Published as a conference paper at ICLR 2025
let Crame
2nd Exama
High Qualitty
Sow Teeporol
Sow Qualitt
High Yesporal
Colverence
XT
XT
X2
X2
X2
X1
X1
X1
Aeselit
Dusvbligg
Deesking
X0
Xg
V0A Xao
X.
The
IV-mixed Sampler (Ours)
Video-based Sampler
Sampler
Figure 2: Overview of our IV-mixed Sampler, Video-based Sampler and Image-based Sampler. /V-mixed Sampler utilizes IDM and VDM to ensure synthesized video quality and temporal coherence, respectively.
present IV-mxxed Sampler (wxzt. "IV-IV"" on Fig. 2 and its pseudo code in Appendix B. 2) The empirically optimal IV-mixed Sampler further reduces the UMT-FVD by approximately by 39.72 points over Freelnit (Wu et al., 2023). Furthermore, 3) we conduct sufficient ablation studies to determine which classifier-free guidance (CFG) (Ho & Salimans, 2021) scale and which sampling paradigm yield the best performance for various metries at what sampling intervals. In addition this, 4) qualitative and quantitative comparison experiments have amply demonstrated that our algorithm achieves state-of-the-arr (SOTA) performance on five popular benchmarks: UCF- 101- FVD (Soomro, 2012), MSR- VTT-FVD (Xu et al., 2016), Chronomagic- Benciic-BBBhh--e 150 (Yuan et al., 2024), Chronomagic- 1649 (Yuan et al., 2024) and VBench (Huang et al., 2024). These competitive outcomes demonstrate that our proposed /V-mixed Sampler dramatically improves the visual quality and semantic faithfulness of the synthesized video.
2 PRELIMINARY
We review VDMs, SDEdit, DDIM & DDIM-Inversion in this section and further bootstrap how past work has designed VDM-based plug-and-play algorithms Appendix A.3.
Diffusion Models. Diffusion models (Ho et al., 2020; Song et al., 2023e;a) including IDMs and VDMs consist of a forward process and a reverse process. Given Xo represent a D-dimensional ran- dom variable sampled from the real data distribution (xo)) The forward process injects Guassian noise 62 ~ (0, I) to the clean data as follows:
X 0,80 + Oefe.
(1)
where t 2/in, 1] (1) is very small quantity defaulting le-5) and a, and de are components of a predefined noise schedule. ag is monotonically increasing from 0 to in all diffusion models, while ar can either remain constant (e.g., EDM (Karras et al., 2022)) or decrease monotonically (e.g., VP-SDE (Song et al., 2023c) and Rectified Flow (Liu et al., 2022)) from 0 to 1. The forward process in Eq. can be rewritten the following stochastic differential equation (SDE):
dx, f())x;dt + g(t
where (t) and gt) denote the drift coefficient of x4 and the diffusion coefficient of x,, respectively. w. refers to standard Wiener process. Eq. 2 has corresponding ODE-based reverse process defined as:
dx; = [(Ixx - 1 (1)
where Vx log qe (x) denotes the score function Vx, log Pt (x:). Since Vx log qu (x) cannot be ac- cessed during the reverse process, it must be replaced by a linear transformation fa(x..)) using the noise estimation model ea(:)) -) or the score function estimation model so())))
3

---

Published as a conference paper at ICLR 2025
Video Diffusion Model vs. Image Diffusion Model. The most critical differences between IDM and VDM lie in their training sets and model architectures. Regarding training sets, IDM typi- cally utilizes large-scale, high-quality datasets, enabled by the increasing availability of both real and synthetic image datasets (Schuhmann et al., 2022). Conversely, the performance of VDMs is significantly constrained by the limited number of publicly available video datasets, such Webvid- 10M (Bain et al., 2021) and Pandas-7OM (Chen et al., 2024c), which contain low-quality real data. In terms of model architectures, several VDMs differ from their IDM counterparts because video data Xo E Rexextxhxw includes an additional time dimension t, unlike image data Xo E R xxxxxxxww where b, c. h, and refer batch size, number of channels, height, and width, respectively. There- fore, most VDMs incorporate both spatial and temporal blocks, where the spatial block aligns with the module used the mainstream IDMs, while the temporal block employs 3D convolutions (Guo et al., 2023) or tokens in the time dimension (Ma et al., 2024) to capture temporal information.
SDEdit. SDEdit (Meng et al., 2021) is an image editing method that produces a latent variable capable of reconstructing the input Xg. It first injects Gaussian noise into Xg. then performs editing in the latent space, and finally obtains pure samples through standard reverse process. This approach provides a scheme for editing the latent variable using IDM at each step of sampling. Since SDEdit is nontrivial to invert, we employ DDIM-Inversonn (Mokady et al., 2023) to map the latent x, back its previous counterpart Xt+Att where 1 2 At
DDIM & DDIM-Inversion. DDIM (Song al., 2023a) serves an efficient ODE-based sampler that iteratively refines the initial Gaussian noise XT, ultimately generating a "clean" video Xn that conforms to the data distribution px())). A notable characteristic of DDIM is its adherence to deterministic sampling. The synthesized data produced by DDIM can be expressed as follows:
X, = L(x.) = at Xx - (x00,6)(xx,))
+ (x),(x,,)).
(4)
Ot
In this equation, and I. denote timesteps E 1/00, 1], with the condition that / S When under the constraint t 2 s, the standard DDIM in Eq. 4 is referred to as DDIM-Inversion x; = L (x,)). In practical, DDIM is often incorporated into classifier-free guidance for more precise control well as higher quality data synthesis. Thus, Eq. 4 can be transferred into
X: _(x.,.) w) [aix, + (a.0r a.w ()c,(X,, s,c) - wes (x., 2)] /a.,
(5)
where e, Ø and w stand for the text prompt, the null prompt and the CFG scale, respectively.
Additionally, our proposed IV-mixed Sampler, along with Z-Sampling (LiChen al., 2025) and to
Golden Noise (Zhou et al., 2024) applied in the image domain, uses DDIM and inject semantic information, which is essentially a noise optimization framework (Qi et al., 2024). Unlike these approaches, which consider how to perform more efficient resampling operations, /V- mixed Sampler focuses on utilizing both IDM and VDM to maximize the visual quality and motion consistency of the synthesized video.
3 APPROACH
Observing that IDMs produce high-quality samples without ensuring temporal coherence, while VDMs ensure temporal continuity but generate low-quality video, we propose IV-mixed Sampler to combine the strengths of both IDMs and VDMs (see Fig. 2). In this section, we first describe how to sample from X₁ to X and from X1+AI to x₁ using IDM and VDM. We then introduce /V-mixed Sampler and outline its design space of hyperparameters, followed by theoretical analysis. Finally, we discuss the effect of sampling in the latent space on IV-mixed Sampler.
3.1 FORWARD (Go!!) AND REVERSE (BACK!!)
A crucial step in overcoming the bottleneck of VDM ey ((--) with IDM c/(-.-) -) is to address the domain gap between IDM and VDM caused by differences in training data. Therefore, specialized rescheduling paradigm is required to achieve the mapping X, + L f-1(xx (xx) + L(E- (xx)) = X,,
where E(-) is deterministic function with inverse L (.). Given this, we can modify X R
4

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison with popular heavy-inference algorithms on Chronomagic-Benche (+2 [+)) stands for the improvement of /V-mixed Sampler compared the "Standard DDIM". 14VGen is compared across other architectures because its official implementation is limited Animatediff.
Method
Extra Model
LMIT-FD
UMTScoree (f)
GPT (f)
VideoCratterV2
Standard +
/
x
214.06
2.77
2.85
/V-mixed Sumpler (ours)
210.58 2: 010 (-A 3:48)
3.29 20.00 (+A 0.51
3,30 20.005 (-1 0.45)
Standard DDIM
X
241.61 10.002
2.65 20.017
2.97 1000s
ModelSccue- T2V
Freelnit
K
220.96 10.002
2.99 10.025
3.10 10:015
[Vimired Sampler (ours)
V
234,90 (-A 6.71)
3.00
0.35)
3,16 #o.oss
0.19)
Animatediff (SD V1.5,
Standard DDIM
X
275. 18 100000
2.81 20.000
2,86 1:0
Freelnit
268.31 10.000
2.85 1..002
2,60 1.0.08
Motion Adapier V3)
14VGen
227.22 +0.010
2.68
=0.030
3.01
/V-mixed Sumpler (ours)
228.66
(+& 46.58)
3.32=0.018
(+A 0.51)
3.55.0.020 (+-B) 0.69)
Table 2:
Quantitative comparison with popular heavy-inference algorithms, including FreeTnit and 14VGen, on Carroommaicc-ench---6e (Yuan et al., 2024). Yellow cells stands for the winner.
Model
Method
Extra Model
(MT
UMTScore (t)
GPTHo-MTScors (t)
VidooCrafterV2
/V-mixed Sampler (ours)
Standard DPM-Solver++
/
172.03 00.00 (+2 6.41)
278.44
3.35 (+ 0.60)
2.75
3.05 e0.000 (+A 0.36)
2.69
ModelScupe: T2V
/Vimixed Sampler (ours)
Standard DDIM
V
X
197,4440.000
(+A) 2.08)
3.06 (+ 0.07)
L0.00s
3.24 (+A 0.08)
n.16
Animatediff (SD V1.5,
Standard DDIM
X
219. 29.0000
3 3.08
2.62+0.010
Freelnit
X 209 60
3 3.08
2.72+0.014
Motion Adapter V3)
/V-mined Sampler (ours)
14VGen
192.72_0.000
206.22 20
26.57)
1.39 CIA 0.31) 3.35 (1A 0.73)
3.20
3.09gm.m
L (x:: without disrupting the standard sampling process. We utilize DDIM-Inversion and DDIM to implement L (-,-) and -(),-), respectively. By introducing CFG, we can define a new paradigm implement IDMs' information injection via the operator G, i.e. semantic information injection:
x = I(x,; Wgo: Woack, G. 80 back) = L(u + C(u), Woas), where u =
(6)
In Eq. 6, wgo and Whuck stand for the CFG scales for the DDIM-Inversion and DDIM, respectively. go (.,)) and back b,,ck(,,)) are the noise estimation models used to perform L- (.,(), and L(,,),, respec- t0
tively. G is any function whose mission to make modifications to u. It is worth noting that for IDMs, we first reshape X from the shape b x e x h x w to (bx I) x e h x w. and then pass it through the noise estimation model to perform the operation I the practical implementation. If Wgo Wback, < (..)) = G (.,-) and G(-) is an identity operator, then x/ X,. To understand how injects semantic information, we rewrite Eq. 7 using a first-order Taylor expansion as
I(xi: Wao wheck c5. 200) L(L (x.. + 6(2 (x., ww))), wask) L(E (xc, (o0)), wge)
+ (Waa wx)) ollc (xx, up): G (x..
OLLL (x,,, O( w)
Owp
0 (x.,ww)
+ Ofa(L ((x,,))) (w)))
xx define a 1 1
(xe [J[( 1) (xetaee At, Whise back
+ (Waxs - wx)
(7)
+ (Wrea wa)) 2 ((ar00tAl aspsteelllu (x0,, wynch (xe, 1,)]a,4444))
Ougg
86(L-1 (x( Wgel
x, + (Whosk (gg) (aio aro (x.. (.)
a(2 a( )
o5 (x;, Wgo)
OL(L-1 (x) won))
+ 0(2 (xx, w
A Ignore awonddardee and higher terms.
(x-1(x Wpo)
Observing the pink term in Eq. 7, we can inject semantic information from both & ( and danck (., into X2 by setting Whack - Wgo > O. In particular, we set Weack = -ygo and Wback > 0 by default. Given this, we only need to replace e with c)(.-) or e (:.-) -) to inject specific semantic information, thereby enhancing the visual quality or temporal coherence of the synthesized video.
5

---

Published as a conference paper at ICLR 2025
340
Ph
KL
220
40 29%
401 100
6% mn
1811
Co yath
0h: 1882
Figure 3: Ablation studies on sampling intervals of TV-mixed Sampler ("IV-IV"") with Animatediff (SD V1.5, Motion Adapter V3). The BEGIN8-END% in the legend indicates the portion of the entire sampling process performed by /V-mixed Sampler. For example, in 50-step sampling scenario, 0%-50% corresponds /V-mixed Sampler being applied during steps 1-25. More details of "IV-VI" can be found Appendix D.
225
235
3.4-
G24
250
200
265
1.5
275
2.8-
200
IOMM IV VENI IVV IVW
B.V vi-0
R6AMRRoe VENT 1VS
B-W BA vV.B
DIEMPINNNI VLIV VENT
WAT IV JVV VVa
Figure 4: Ablation studies different C species combinations with Animatediff (SD V1.5, Motion Adapter V3). We can clearly observe that IV-mixed Sampler ("IV-IV") is the winner across all metrics.
3.2 IV-MXXED SAMPLER
The paradigm defined in Eq. 6 enables us to casily describe all the base Operators in IV-mixed Sampler. We define "R-", "I-", and "V-" as follows: /) "R-": Forward noise addition using random noise; 2) "I-": Forward noise addition using DDIM Inversion with IDM; 3) "V-": Forward noise addition using DDIM Inversion with VDM. Moreover, the front of the horizontal line "." refers to the additive noise form, while the back of "_" represents the denoising paradigm. For instance, "I-I" and "V-V" can represent [xx, -h, h, G, c/. </) and I(x,, - hh, h, G, E e respectively, under the condition h > and G is an identity operator. Clearly, this represents only single-step injection of semantic information. We can construct /Vmixed Sampler to allow multi-step injections of semantic information using G and the recursive definition:
x = d' (x,) = I(x,, - -hh h, d² co l.buck
Gy = I(y, -h, h, G3 EG
2.80 2.back
(8)
G (y) I(y, NN1 -g
N+1
Nogo
back
s.t. N21
where N and N-1 refer the number of semantic information injection and the identity oper- ator, respectively. Through Eq. 8, we can easily represent /V-mixed Sampler with different N. For instance, "IV-VI" can be described as x (x,) I(x,, -h. h, G c/). d2 I(y, h, G E e and q is an identity operator. Considering the computational overhead and performance trade-offs, this paper focuses only on the scenario where N = 2. In Sec. 4's ablation 2aba to two models each, occupied by e/(,,) and cy (:.-) (wEE. CJ combinations), and find that "IV-IV" performs best. The visualization in Fig. demonstrates that "IV-IV" significantly improves both the visual quality of the synthesized video and the consistency between the video and the text prompt.
ba 2.go study, we further restrict 2..80, 66 6n and 6Ø
3.3 DISCUSSTON
Hy Hyperparameter Design Space. In this paper, we elucidate three design choices: the C² combi- nations mentioned in Sec. 3.2, the intervals at which IV-mixed Sampler performed during standard

---

ODE. For Published as a conference paper at ICLR 2025
Nermakodd
Droxnis Suale
VIAM Dreacali
Figure 5: Ablation studies on dynamic CFG scale with Animatediff (SD V1.5, Motion Adapter V3). We ensure that It = 1/ (l). Consequently, the IDM dynamic CFG scale corresponds to IDM's %/p0 7go Similarly, the VDM dynamic CFG scale corresponds to VDM's 1.- --1 logr(p) is I, O, and -1 indicates "go
whether the change from "/g to .El is convex, linear, or concave, respectively. All metrics are normalized. Note that we back-normalized UMT-FVD for consistency.
DDIM sampling, and the dynamic CFG scale. All three forms expected to find the empirically optimal solution. with the first two being natural explorations and the last addressing the domain gap between IDMs and VDMs, which is expected be mitigated by adjusting the CFG scale. For the last one, to be specific. we re-express -h and h in Eq. 8 as (t) and h backk))). (t). Inspired by the Karras's noise schedule (Karras al., 2022), we define (1)(()) and back (1) as
(1) = ((x0d)
m-
1-10 )39, ( (1) = 1-11
t( +
c-ll (Tucc
(9)
where 7 and buck represent the CFG scales when f 1, while 2/50 and Tback denote the CFG scales when = O. The parameter controls the concave and convex properties of the CFG scale curve with respect The experiments in See, demonstrate that the dynamic CFG scale can be adjusted to achieve performance improvements for specific aspects (e.g., semantic faithfulness).
Theoretical Analysis. shown in Theorem 3.1, /V-mixed Sampler can be elegantly transformed into an ODE, taking the same form as Eq. 3. Consequently, IV-mixed Sampler preserves the stan- dard sampling process (e.g., DDIM or MallermMrryy,a), enabling a trade-off between temporal coherence and visual quality by adjusting the parameters WIDM w VDM
and w.
go-backk
Theorem 3.1. (the proof in Appendix C) IV-mixed Sampler can be transferred example, the ODE corresponding to "IV-IV" is
1 - 92)())
g2(1) V log di (c)x) (w (w-backk + w) g2(1) Vx log VDM VDM (c|x)
dx: f(t)x
Ro-back
(10)
Here, w refers to the vanilla CFG scale, while both JDM back and go-bbcc are CFG scales that are greater than O. Let x log I!M (clx) and Vx log qf YDM (c|x) represent the score function estimated for (x() using IDM and VDM under classifier-fre guidance.
The Influence of Latent Space. Most current high-resotution IDMs and VDMs follow the la- tent diffusion model (LDM) paradigm (Rombach et al., 2022a). As a result, leveraging IV-mixed Sampler requires IDM and VDM to share the same latent space, meaning they must use the same VAE (Kusner et al., 2017). 7). Fortunately, for most VDMs, we always can find the corresponding IDM that share the same latent space with the VDM. For cases where this condition is not met, can convert the video from shape excxtxhxxx to (bxt)xcxl xhxw and then pass it through VDM, which is referred using "IDM". This paradigm is also applied in I4VGen (Guo et al., 2024).
EXPERIMENT
In this section, we present experiments demonstrate the effectiveness of IV-mixed Sampler. Specifically, we perform qualitative and quantitative comparisons across various benchmarks on multiple T2V diffusion models, including ModelScope--T2 (Wang al., 2023), Animatediff (Guo

---

Published as a conference paper at ICLR 2025
Table 3:
Quantitative comparison on UCF-101 datasets using Animatediff and ModelSeope-12V. (+A
F) indicates the improvement of /V-mixed Sampler compared to the "Standard DDIM".
CV)
Model
Method
Extra Model
(StyleGAA)
(VideoCiPT)
Standard DDIM
X
+0..07
819.93+0.015
Animatediff (SD V1.5.
FreeInit
X
20.033
807.02 po.002
Unictel
X
1859. 11 gg
1863.32 1o.0ss
Motion Adapter V3)
14VGen
/
803.26
805.75_o.0
Wamiised Sampler (ours)
/
800.47 [1.7_0000 (e 14.60)
804.67 10.00s (HA 15.26)
ModeIScope- T2V
Standard DDIM
x
1492. 16
1484,98
IV-mised Sampler (ours)
841 841.65 (+D 650.51)
838.05=0.01 (4. 656.93)
Table 4: Quantitative comparison on MSR-VTT using Animatediff and ModelScope-T2V. (+A [D)
indi-
cates the improvement IV-mixed Sampler compared to the "Standard DDIM".
CVD
FVD (VidzoGPT)
Model
Method
Extra Model
(StyleGAN)
Standard DDIM
762.2210.0ra
761 200o
Animatediff (SD V1.S.
FreeInit
732.59
731 23..000
Motion Adupter V3)
J4VGen
/
741 741.552000:0
740. 20.000
IV-mised Sampler (ours)
/
721.31 o11 [-A 40,91)
719.92 40.001 (+-441.2 41.12)
Standard DDIM
739.13
787.231o.or
WV-mived Sampler (ours)
600.31 (+A 135.82)
601.88_0.01 (FAA 135.35)
al., 2023), and VideoCrafterV2 (Chen et al., 2023a). Additionally, we compare IV-mixed Sam- pler with three heavy-inference algorithms, FreeInit (Wu et al., 2023), Unictrl (Chen et al., 2024d) and I4VGen (Guo et al., 2024), on VDM. Note that I4VGen is also an algorithm designed to en- hance VDM performance using IDM. However, its IDM is configured to be consistent with VDM, achieving image synthesis by reshaping from bxcxtxhx to (bxt)xcx xxhw before passing it through VDM. For further details, please refer to Appendix A.3, Finally, we conduct extensive ablation studies and present visualization to validate the optimal solution of various design choices. More implementation details can be found in Appendix A.
4.1 MAIN RESULTS
Chronomagic-Bnnch 150 & 1649. We evaluate the effectiveness of IV-mixed Sampler on three different VDMs: VideoCrafterV2, ModelScope- T2V. and Animatediff (SD V1.5, Motion Adapter V3). The comparison results on Chronomagic--enchh-155 (w.r.1, 150 prompts) are presented in Table 1. We employ three metrics in this benchmark: UMT-FVD (for visual quality), UMTScore (for semantic faithfulness), and GPT4o-MTScore (for temporal coherence and metamorphic amplitude) to assess our proposed method. The experimental results in Table show that IV-mixed Sampler significantly outperforms both the standard sampling method and other computationally intensive algorithms. This indicates that rational integration of IDM and VDM has great potential to improve the performance of video synthesis tasks. Although /V-mxxed Sampler does not outperform all comparative methods on Animatediff and ModelScope-TTV in terms of UMT-FYD, it still shows a significant improvement over standard sampling algorithms. It is worth noting that ModelScope- T2V was unable to locate a high-quality IDM due to its low resolution (i.e. 224x224) synthesized video. In contrast, /V-mixed Sampler do not perform best on Animatediff because 14VGen is an algorithm that integrates both IDM and VDM. Moreover, on the more comprehensive Chronomagic- Bench- 1649 (w.r.t, 1649 prompts), /V-mixed Sampler outperforms all comparative methods across all metrics and models. As illustrated Table 2, /V-mixed Sampler achieves the best performance on all metrics, even when compared to the latest SOTA method, 14VGen. These experimental results underscore the strong generalization capabilities of /V-mixed Sampler, highlighting its effectiveness as plug-and -lay solution that can be seamlessly integrated across various VDMs.
For comprehensive evaluation of both visual quality and semantic consistency, we fur- ther assessed the performance of IV-mixed Sampler on VBench (Huang et al., 2024), with the results presented in Table 5. From Table 5, it is clear that /V-miedd Sampler outperforms vanilla sam- pling across most metrics, particularly on the multiple objects, where IV-mixed Sampler improved the performance of Animatediff from 36.88% to 58.46%. Additionally, the average scores of /V-
VBench.
8

---

Published as a conference paper at ICLR 2025
Table 5: Quantitative comparison on VBench (Huang et al., 2024), (+ A
represents the improvement of
IV-mixed Sampler compared to Vanilla Sampling.
Model
Mehhol
Aoorage
Sutjes Urmootzzcy
Temponl
Chrest
Mattpl
Huraa
Antratediff
(5D V1.5.
Vasili Scrpttng
60, 19.99%
95, 30.
98 75%
90 90.90
36,80%
92
Mokinn Albperr Vi2
Friativl Stepler
6560%
91.11%
47,0056
W. NPS (ei ( (WN))
59.4%%
90W0%
WOOKS
Mistekespe-122
Promtiall Sampler
Vavila Sanyling
57,8%) (U5499)
57.35%
88,39%
98.0%
82.25%
79442
36,15%
38,08%
94.00%
91,406
2,2069)
Model
Mothod
Coker
Spuria
Nooee
Agpai W933 Style
Irineral Siyle
Theral Constorry
VL.,
Vasili Sumpling
$7,47%
34,50%
50.19%
22,42%
26/13
27,046
Matia Alkiier V2)
(Fantiaal Sampler
91306
9A 78/% (1:a B8%)
56.9%%
24.49% (1:0) 20122))
27 62N
(.99%)
29.54%
(9r4 235053
Masthkcspo-TV
Vasill, Sccppiig
35,42%
33,68%
42.02%
(110)
25
25,37%
45.
25,676
mixed Sampler exceeded those of vanilla sampling on both Modelscope- T2V and Animatediff, fully demonstrating the effectiveness of /V-mixed Sampler.
UCF-101-FVD & MSR-VTT-FVD. Additionally, to complement our experimental results, we conduct experiments on several traditional benchmarks, including UCF-101-FVD and MST- FVD. The FVD results for UCF-101 and MSR-VTT are presented in Table 3 and Table 4, respec- tively. These quantitative results significantly substantiate the superiority of IV-mixed Sampler, high- lighting its effectiveness in outperforming existing seavy-inference approaches these results and visualized in Fig. I, /V-mixed Sampler has strong generalization ability across different VDMs, which possesses significant practical application value in real-world scenarios.
VDMs. Through
4.2 ABLATION STUDIES
We begin with basic ablation study to demonstrate the validity of inserting IDM into the denoising process of VDM. We present the experimental outcomes in Fig. 6, we can discover that the approach "R-I-I", which use Gaussian noise to perform the forward diffusion process, result in significantly lower quality of the synthesized video compared to the standard DDIM process (ie., Origin in Fig. 6). This phenomenon arises because "R-[-"" over-introduces invalid information (i.e., Gaussian noise) into the synthesized video during denoising. Thus, we consider the more robust DDIM- Inversion integrate the video denoising process with the image denoising process, as this paradigm is stable and effectively reduces truncation errors in practical discrete sampling.
After that, as described in Sec. 3.3, we elucidate three design choices, namely the sampling interval of /V-mixed Sampler, the C²¹ species combinations, and the dynamic CFG scale. For 1) the sampling interval of /V-mixed Sampler, the results Fig. 3 clearly illustrate that performing /V-mixed Sampler across all sampling steps is optimal. Another evi- dent conclusion is that the closer the sampling in-
RK-II
UMTScoze:
RR-VV
*
UNT-FVD:
Vinsad quality
R-1
5.75
*
Origin
I1-11
VEAV
terval of /V-mixed Sampler is to t 1, the more sig- nificant the performance gain. This suggests that to save computational overhead, IV-mixed Sampler can be applied within the 0%-50% sampling inter- val or even restricted to the 0%-25% interval. For 2) the C7 species combinations, we present its ab- lation results in Fig. 4. It is evident that "IV-IV" outperforms all metrics and significantly surpasses both FreeInit and standard DDIM. This suggests that there is an empirically optimal combination of results within IV-mixed Sampler. Accordingly, we use "IV-IV" for all comparison experiments. For 3) the dynamic CFG scale, the conclusions are not as intuitive as the first two design choices. Specif- ically, we considered total of 7 combinations of
+1.00
w-V
-1.25
-1.50
21.25
-2.00
830
100
730
200 250
245 243 208 235
LM
Figure 6: UMTScore (1) vs. UMT-FVD (L) with Animatediff (Guo et al., 2023) on Chronomagic- Bench-150 (Yuan ct al., 2024). In the legend, "R", "I", and "V" represent the score function estima- tion using random Gaussian noise, IDM, and VDM, respectively. Moreover, the front of the horizontal line "_" refers to the additive noise form, while the back of "_" represents the denoising paradigm. For instance, "RR-II" stands for a two-step of adding noise with Guassian noise followed by two-step of denoising performed using IDM.

---

Published as a conference paper at ICLR 2025
dynamic CFG scales, where the CFG scale varies from 2 to 7go across IDM dynamic cfg scale, VDM dynamic cfg scale and p. In Fig. 5, we set p to 7, and 1/77 model changes in the CFG scale as convex, straight, and concave functions, respectively. For the specified
1, we consider goo-back and VDM's CFG scale wgo-buck' (1) remaining constant, go-back decreasing. For the "constant" case we make for the "decreasing" case make 7go 6, 7go 2. and for the "increas- ing" we make 1 2, ا- = As illustrated in Fig. 5, we find that for visual quality (w.rt. VDM
3 types of the IDM's CFG scale I.DDM
(2) w ggo-back/ go-back increasing, (3) who-backk "go-back /w VDM
IDM /w
7go
'go
UMT-FVD), keeping both w ggo-back and w go-bbcc constant is the best choice, For semantic faithfulness (w.rt., UMTScore), the optimal strategy is have both WiD and W.DM go-hack increasing. For tempo- w VDM
ral coherence (w.z,, GPT40-M -.ooore),,,,, it is optimal for wDI ggo-back to decrease while "go-back increases. Therefore, to enhance the performance of a specific aspect of synthesized video, we can adjust the dynamic CFG scale to achieve the empirically optimal trade-off.
Visualization. We visualize the standard sampling and /V-mixed Sampler of the synthesized video Fig. 1. It can be observed that TV-mixed Sampler significantly improves both visual quality and semantic faithfulness. In addition to this, we empirically invited a number of other AIGC-related researchers to judge the video quality and agreed that IV-mixed Sampler's enhancement could be observed by the naked eye.
5 LIMITATION
Althrough /V-mixed Sampler significantly improves the performance of VDM, it introduces addi- tional computational costs. For "IV-IV" on Animatediff, increases the number of function evalu- ation (NFE) from 50 to 250. In the practical implementation, the computational overhead went up from 21s to 92s at a single RTX 4090 GPU. This problem could potentially addressed in the fu- ture by distillation algorithms similar to accelerated sampling (Song et al., 2023b; Shao al., 2023; Salimans Ho, 2022). This exploration of INFERENCE SCALING LAWS first, and then distilling the performance gains it achieves back to the foundation model may be viable path for the future.
CONCLUSION
In this paper, propose IV-mixed Sampler enhance the visual quality of synthesized videos by leveraging an IDM while ensuring temporal coherence through a VDM. The algorithm utilizes DDIM and DDIM-Inversion to correct latent representations x, any time point I, enabling seam- less integration into any VDM and sampling interval. /V-mixed Sampler can be formulated as an ODE, achieving a trade-off between visual quality and temporal coherence by adjusting the CFG scales of both the IDM and VDM. In the future, we plan to fine-tune several stronger IDMs, such as FLUX, to better adapt the latent space of target VDMs, thereby further enhancing the performance of VDMs. We anticipate IV-mixed Sampler will be widely applicable in vision generation tasks.
ACKNOWLEDGE.
This work was supported by Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B 121 12010000..
p 7 is the default implementation of Karras's 's noise schedule (Karras al., 2022).
10

---

Published as a conference paper at ICLR 2025
Ethics Statement. We present /V-mixed Sampler, method designed to enhance the semantic accuracy and visual quality of video produced by Video Diffusion Models. Although our approach does not directly engage with real-world datasets, prompts, while respecting user autonomy and striving for positive outcomes. Acknowledging the commercial potential of /V-mixed Sampler, emphasize responsible and ethical deployment of the technology, aiming to maximize societal benefits while carefully mitigating any potential risks.
dedicated to ensuring the ethical use of
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021.
James Betker, Gabriel Goh, Li Jing, + TimBrooks, Jianfeng Wang. Linjie Li, t LongOuyang, JuntangZhuang, f JoyceLee, & YufeiGuo, WesamManassra, & PrafullaDhariwal, CaseyChu, YunxinJiao, and Aditya Ramesh. Improving image generation with better captions. 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprinl urXiv:2311.55127 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exciee Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.1999 1955,, 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024a.
Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. arXiv preprini arXiv: 15241, 2023b.
Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Your diffusion model is secretly a certifiably robust classifier. arXiv preprint arXiv:2402.02316 2024b.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wet Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024c.
Xuweiyi Chen, Tian Xia, and Sihan Xu. Unictrl: Improving the spatiotemporal consistency of text-to-video diffusion models via training-fres unified attention control. arXiv preprint arXiv:2403.02332, 2024d.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang. Ming-Yo Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 019930229941, 2023.
Xiefan Guo, Jinlin Liu, Miaomiao Cui, and Di Huang. 14vgen: Image as stepping stone for text-to- video generation. arXiv preprint arXiv:2406.005 02230, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
11

---

Published as a conference paper at ICLR 2025
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXi:2104.08718, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Neural Information Process- ing Systems Workshop, Virtual Event, Dec. 2021. NeurIPS.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems, pp. 6840-6851, Virtual Event, Dec. 2020. NeurIPS.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565 265777, 2022.
Matt J Kusner, Brooks Paige, and José Miguel Hermández-Lobato. Grammar variational autoen- coder. In International Conference on Machine Learning, pp. 1954-1954. PMLR, 2017.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficien video foundation models. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19948-19960, 2023.
Bai LiChen, Shitong Shao, zikai zhou, Zipeng Qi, zhiqiang xu, Haoyi Xiong, and Zeke Xie. Zigzag diffiusion sampling: Diffusion models can self-improve via self-reflection. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ /ttp:////ppereeeee//mmmm net/Eorum: id=MkvOHleke.
Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Zhiqiang Xu, Haoyi Xiong, James Kwok, Sumi Helal, and Zeke Xie. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint wxXv:2440.072533, 2024a.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022.
Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. Ad- vances Neural Information Processing Systems, 36, 2024b.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint ArXiv:2401.03044 2024.
Mage.Space. Realistic vision model, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 10Xiv:2/08.0107: 2021.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296- 4304, 2024.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2025
Zipeng Qi. Lichen Bai, Haoyi Xiong, and Zeke Xie. Not all noises created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407.14041, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition (CVPR), pp. 10684-10695, June 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjÅmjm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In In- ternational Conference on Learning Representations, Virtual Event, Apr. 2022. OpenReview.net.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294 2022.
Shitong Shao, Xu Dai, Shouyi Yin, Lujun Li. Huanran Chen, and Yang Hu. Catch-up distillation: You only need to train once for accelerating sampling. arXiv preprint arXiv:2305. 10769, 2023.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations, kigali, rwanda, May. 2023a. OpenReview.net.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint 269,002300..116,, 2023b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, kigali, rwanda, May. 2023c. OpenReview.net
Soomro. Ucf101: dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv:1212.0402, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13377, 2021,
Thomas Unterthiner, Sjoerd Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski. and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Mod- elscope -ot-too-iieo technical report. arXiv preprint arXiv:2308. 06501, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Frecinit: Bridging initial- ization gap in video diffusion models. arXiv preprint arXiv:2312.07553) 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296. 2016.
Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-bench. A benchmark for metamorphic evaluation of text-to-time-lapse video generation. arXiv preprint arXiv:2406. 18522, 2024.
Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, and Lijuan Wang. Motion consistency model: Accelerating video diffusion with disentangled motion aption-appearance distillation. arXiv preprint 4rXiv:2406.0688) 2024.
13

---

Published as a conference paper at ICLR 2025
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. arXiv preprint arXiv:2411.0950) 2024.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 BENCHMARKS
We present the relevant metrics and benchmarks used for comparison in the main paper..
UCF-101-related FVD. The UCF-101 dataset is an action recognition dataset comprising 101 categories, with all videos sourced from Youtube. Each video has fixed frame rate of 25 frames per second (FPS) and resolution of 320x240. Several previous works (Blattmann et al., 2023; Wu et al. 2023; Chen et al., 2024d) have validated the generation performance of VDMs on the 1CF dataset using Fréchet Video Distance (FVD) (Unterthiner et al., 2019). However, comprehensive evaluation benchmark for UCF-101 is still lacking. To address this, we follow the methodology of FreeInit, utilizing the prompts listed in Ge et al. (2023) to synthesize videos and assess inference performance with FVD. Specifically, we synthesize 5 videos for each of the 101 prompts provided by Ge et al. (2023), resulting in total of 505 synthesized videos. We then compute the FVD between these 505 synthesized videos and 505 randomly sampled videos from the UCF-101 dataset (5 per class), using the built-in FVD evaluation code from Open-Sora-Plars
MSR- -T--elat FVD. The MSR-VTT dataset (Xu et al., 2016) is large-scale dataset for open-domain video captioning, featuring 10,000 video clips categorized into 20 classes. The stan- dard split of the MSR-VTT dataset includes 6,513 clips for training, 497 clips for validation, and 2.990 clips for testing. For our evaluation, we utilize all 497 validation videos. To ensure evaluation stability, we synthesize a total of 1,491 videos based on prompts from these validation videos, with each prompt producing 3 different videos. We assess the results using the built-in FVD evaluation code from Open-Sora-Plae
Chronomagic-Benete Bench-150. Chronomagic-Benchh 150, introduced in (Yuan et al., 2024) and re- cently accepted by NeurIPS 2024's dataset and benchmark track, serves as a comprehensive bench- mark for metamorphic evaluation of timelapse T2V synthesis. This benchmark includes 4 main categories of time-lapse videos: biological, human-created, meteorological, and physical, further divided into 75 subcategories. Each subcategory contains two challenging prompts, leading to in a total of 150 prompts. We consider three distinct metrics in Chronomagic Bench- 150: UMT-FVD (J), UMTScore (1), and GPT4o-MTScore (1), each addressing different evaluation aspects. Specifi- cally, UMT-FVD (L) (Liu al., 2024b) leverages the UMT (Li et al., 2023) feature space to compute FVD, assessing the visual quality of the synthesized video. UMTScore (1) utilizes the UMT (Li et al., 2023) feature space to compute CLIPScore (Hessel et al., 2021), evaluating the text relevance of the synthesized video. Lastly, GPT4o-MTScooe (1) is fine-grained metric that employs GPT- 4o (Achiam et al., 2023) as an evaluator, aligning with human perception to accurately reflect the metamorphic amplitude and temporal coherence of T2V models.
introduced in (Yuan et al., 2024) and re- cently accepted by NeurIPS 2024's dataset and benchmark track, is a comprehensive benchmark de- signed for the metamorphic evaluation of timelapse T2V synthesis. While it shares 75 subcategories with Chronomagic- Beeee-Bncc-lle 150, it offers a more extensive evaluation framework with 1649 prompts, making it significantly more comprehensive than its lightweight counterpart Chronomagic-Bench- 150. Chronomagic-Bencce -64ch-166nnn-- includes 4 key metrics: UMT-FVD (L), MTScore (t), UMTScore (1), and GPToc-MTScore (f), each serving to evaluate different aspects of video synthesis. Specifi- cally, UMT-FVD (L) (Liu et al., 2024b) utilizes the UMT (Li et al., 2023) feature space to compute FVD, assessing the visual quality of the synthesized videos. MTScore (1) measures metamorphic amplitude, indicating the degree of change between frames. UMTScore (f) leverages the UMT (Li et al., 2023) feature space compute CLIPScore (Hessel et al., 2021), evaluating the text relevance of the synthesized videos. Finally, GPT40-MTSScm (1) is a fine-grained metric that employs GPT- 4o (Achiam et al., 2023) as an evaluator, aligning with human perception to accurately reflect the metamorphic amplitude and temporal coherence of T2V models. As with to Saanommaic-BBnnhhh 150, we choose ignore the MTScore (f) metric in our experiments due its limitations.
Pronomaaicc-Bnchhh
Chronomagic-Bendee
phttps://gt thub. com/ http://KKK-Yuanrrroo////eeee
Plan
15

---

Published as a conference paper at ICLR 2025
A.2 VIDEO DIFFUSION MODELS
We describe the VDMs utilized in this work. Specifically, we employ 3 VDMs with distinct architec- tures: ModelScope-T2y (Wang et al., 2023), Animatediff (Guo al., 2023), VideoCrafterV2 (Chen et al., 2024a).
ModelScope-T24 ModelScope-TZ2 incorporates spatio-temporal blocks to ensure consistent frame generation and smooth motion transitions. Its key features include the utilization of 3D con- volution andtraining from scratch. The input video size is structured as 3 x 16 256 x 256, where 3 represents the number of channels, 16 is the number of frames, and 256x256 indicates the resolution of each frame. This configuration allows the model to effectively capture both spatial adn temporal features, facilitating high-quality video synthesis.
Animatediff. Animatediff does not require training from scratch; instead, it only needs fine-tuning on existing image diffusion models. Its motion adapter serves as a plug-and-plae play module, allowing most community text-to-in text-to-image models to be transformed into animation generators. In this paper, we consider the latest version Animatediff (SD V1.5, Motion Adapter V3), which is fine-tuned from SD V1.5. The input video size of Animatediff (SD V1.5, Motion Adapter V3) is 3 16 512x 512, where 3 indicates the number of channels, 16 represents the number of frames, and 512x112 specifies the resolution. Note that there are differences in the performance of this VDM because we used different resolution than the one used in the Chronomagic-Bert
paper (Yuan et al., 2024).
ViaeoCrafterV2. VideoCrafterV2 focuses on T2V synthesis, aiming to synthesize high-quality videos from prompts. This work investigates training scheme for video models based on Stable Diffusion (Rombach et al., 2022b), exploring how to leverage low-quality videos and synthesized high-quality images to develop superior video model. The input video size of VideoCrafter V2 is x 16 512 x 320, where 3 indicates the number of channels, 16 represents the number of frames, and 512 320 specifies the resolution.
A.3 HEAVY-INFERENCE ALGORITHM ON VDM
Here discuss two popular VDM- heavy-inference algorithms FreeInit (Wu et 2023) and I4VGen (Guo et al., 2024).
I4VGen. I4VGen (Guo et al., 2024) is a training-free free and plug-and-pla video diffusion infer- ence framework that enhances text-to-video synthesis by leveraging robust image techniques. To be specific, 14VGen decomposes the process into two stages: anchor image synthesis and anchor image-guided video synthesis. A well-designed generation-selection pipeline is used to create visu- ally realistic and semantically faithful anchor images, while score distillation sampling (SDS) (Poole et al., 2023) is employed to animate the images into dynamic videos, followed by video regener- ation process to refine the output. In its official implementation, both phases are realized by VDM, where the anchor image synthesis is performed by merging the time dimension into the batch size dimension through VDM. In essence, I4VGen does not introduce true IDMs to improve the quality of synthesized video obtained from VDMs.
FreeInit. FreeInit (Wu et al., 2023) is novel inference-time strategy designed to enhance tem- poral consistency in video generation using diffusion models. This approach addresses a key issue: the difference in the ppatial-temporal frequency distribution of noise between training and inference, which leads to poor video quality. FreeInit iteratively refines the low- cow-frequeccy components of the initial noise during inference, bridging this gap without requiring additional training.
A.4 HYPERPARAMETER SETTINGS
For all comparison experiments, we used the form "IV-IV" and perform IV-mixed Sampler at all 1=0 -n nl and I- all are set as 4. time steps of the standard DDIM sampling. In addition, ?go Thack+ 1go For both Animatediff and ModelScope-T2V, that we experimented with using Mini SD as the IDM for ModelScope-T2V to maintain a consis- tent resolution of 256x256. However, as illustrated in Table 6, we found that its performance was
stable diffusion (SD) V1.5 as the IDM. Note
16

---

Published as a conference paper at ICLR 2025
Table 6: Ablation studies with Modelscope-I2V on Chronomagie-Bnnch--55..
Model
Method
Extra Model
(MT-FVD
UMTScore (1)
GPT4o-MTScore (f)
Standard DDIM
241.61
2.66
2,96
ModelScope-T2V
IV-iisdd Sompler (Mini SD) [V-mized Sampler (SD V1.5)
247.99
2.63
3.00
234.90 (+A 6.71)
3.02 (+-3 0,36)
3.14 (=1) 0.18)
Table 7: Performance comparison of VidcoCrafterV2 across different =% settings.
Model
2%
UMT-FVD (1)
UMTScore (f)
GPT4o-MTScore (1)
Standard DDIM
214.06
2.76
2.87
33.3%
212.74
3.08
3.02
VidcoCrafter
50.0%
208.67 (.39)
3.23
3.18
66.7%
210.57
3.29
0.53)
3.30 0.43)
75.0%
211.87
3.28
3.28
FSG 30
1300
14-2005
15-180%
1200
Figure 7: Ablation studies sampling intervals of /V-mixed Sampler ("VI-IV""). The BEGIN BEGIN%-EE %-END% in the legend indicates the portion of the entire sampling process performed by IV-mixed Sampler.
6,
3.0
4.5
4.5
4.0
3 4.0
3 4.0
3.5
3.5
3.5
3.0
3.0
3.0
2.5
2.5
2.5
2.0
2.0
750
Timestags
Timestops
Timestops
Figure 8: The visualization of Karras's noise schedule.
inferior to using SD V1.5 with upsampling and downsampling. For VideoCrafterV2,, we use Re- alistic Vision V6.0 BI Space, 2023) as the IDM to accommodate a resolution of 512x320. For the remaining configurations, vDMs. Furthermore, we find that applying "TV-IV" at every step on VideoCrafterV2 destroys tem- poral coherence. Therefore, replace "IV-IV" with "VV-VV" for z%. The results of the ablation experiments are shown in Table 7. We finally chose z%=66.7 as the final solution.
follow the sampling form recommended by the corresponding
B PYTHON-STYLE PSEUDO CODE
We present the python-style pseudo-code in Fig. to make it easier understand /V-mixed Sampler.
17

---

Published as a conference paper at ICLR 2025
Algorithm Pseudo code of IV-mixed Sampler (w.rt. "IV-IV") (>$) in PyTorch-like style.
X_T: the initial noise
the number of serpling
schodilezs schedelez sinilar so that or distusers
iny st..cheeeeerri en inversion schecaler similac to tast or ditcucers
idn, F. vons the backbones of TDM and VoM anbodcing IDM VON
po_ian, pe_rdd: the
w_idn_begany K_ion_end, -_vin_negin, v_var_ead, the ayperpazonater of the dynenic CPO scale
def yor_stomas_karraedda signa_begin, siona_end, ston7.0) ramp torca.iinspace (0, b)
begin_inv_são aigna_beqin v* [1 zho)
end_iny_the e sigma_end e* (2 =ho)
signas ibegin_iny_rho
zanp iand_iny_sho bogin_iny__hh)) e the
rcturn sigmas Constructs the neise schodcle of Karras it al.
def sampling achaduler, inv_schedaies, F_idi,
s*wwarqs):
pecicm, so_vdr. joorgg.get ("so_ida", None), keeggg.ecti"se_vdt Renc)
v. begii, s_idiuand, bevinn a_vdn_aad, sho oatros.cat.i (atiiss.sst."_iiin___aae (""_idn_ O) kvargs. gst ("v "u_iddeend" 4), lvergs. ("v_yda_end",
(""_ydn_braia",, kvargs.ges ("tha", N ketrgs.qet ("W", ("8",, 501
18 get_ karzas (N, baidi, siidrend, chot Dynanic CFG acall vár_CFG get__ignna
vevar_becia, MLydinend,
for step in sange [8]
acheduler. Limeatte [step]
min(t schadcler. www.ta...nntooinnnteoo..
achedular. tin..nnnnnrriiiiiii innestea-11
mintt a schedglar.confie cow...g.....m.........m. mww.m..enninnnieeee....
scheduler. inw_infeeeoos_dllee
achodular. contulrrr..ototaae con.io. tnnn 4 Perforn seepl ing wish TBM
latent winops. rears aartaaggeeaaae (latent, "b b.b. (b e
falatent
([latent]
Ialatent schedoler. sendelenndee___gee In_lacent, b)
nupsed = F_ida Iidntaaaltend L, per ids)
Ln, r_pzed_te c.urke.chunk122
rupred s_pred_un con_cPG[stepl ir_pred un
latent ino_scheduler. aaep (nupeed, pat, latenc)
latent = eiineesrrearanee (latent, "Ib t) h w b btst, teferd 4 Perforn sampling wish VEN
fglatent tozchreat ([latent]
latetnt schedoler.scee model. input In lacent, p.t)
rupred = Paadn(n__ltent, Buly pe_vda)
un, pepred_ie
chunk (2)
prered p_pred on crG[stepl
pred s_pred
latent inv inv_scheddler, are in paut, latent)
4 Pectorn sampling with IDM
malatent schedoles pnweeae Inglavent, poLL)
latent = schedo'l cedduler.etee errestepin pred,, pp.t, latenc) 4 Derforn sampling wich VEN
Ialatent achedoler.ace" reunddel Enppt lacent,, Put)
latent schedules ser.dteeeeeeeeeeee pred, Istert]
sarpling VON
n. latent schedoler wew.eerrrcceeeeeeeeeeeeeeeeeeeee input In. lacent, t)
lateat sereecleeeeeeptnp_ppp t, latent) return decode latent (latent)
F transfer latent to video
IV-mixed Samplor
video Ox scheduler, scv_scheduler,
C THEORETICAL PROOF
Here we give the proof of Theorem 3.1 in the main paper. We use "IV-IV" for an example. and the derivation of other forms of /V-mixed Sampler is similar "IV-IV" and is not described additionally. First, /V-mixed Sampler (w.r.1., "IV-IV") can be rewritten as
(t) V x log di
g (t 1) Vx log aetn
x-
X₁
w00
IDM (D
w
VDM (c|x)
2
(11)
(t 1))
(t)
Vx log (clx) whick
V log de VDM (clx),
back
2
2
18

---

Published as a conference paper at ICLR 2025
w/ disei22 Vaw
w/ distil2
VON
Latent
Latent
Latent
times top
timedd top
Eines cap
tines top
times cap
999
759
499
269
Inversion
200
VON
DOIM
w/o distill
VOM
DOUN
w/o distill
VOM
DDIM
Ia bant
Latent
ZDM
Eimeateng bap
timeatap
739
779
Figure 9; Overview of our IV-mixed Sampler the motion consistency model (Zhai al., 2024),
Table 8: Quantitative comparison of motion consistency model (MCM) on Chronomagic-Rench--bba (+A F)) stands for the improvement of /V-mixed Sampler compared to the "Standard LCM"
Model
Method
Extra Model
LM
UMTScore (1)
GPT4e (1)
Standard LCM
277 -0.09
2.66
(Animatediff V2)
Waannedd Sampler (ouns)
246.59 g..001 (1 31.32) 2.19 (1 0.21 2.87_00000
0.21)
where represents the sampling step, which is usually extremely small. Vx log qt (c|x) and vx log ot VDM (c|x) represent the score function estimated for px(x.) using IDM and VDM under classifier- free guidance. Since 7) is very small, and assume that w is larger than who and waeck is larger than 2 Eq. 11 be rewritten as
g²(1)
VDM g²(1) IDM (c(x)) Vx log St
w go-bback
IDM
wgo-back 2
Vx log 4:
VDM (c|x),
(12)
where "go-back and w go-bacck are large than 0. Given this, IV-mixed Sampler can be written as an ordinary diffcrential equation:
dx: f(()x 1 -0)(0)
2²(1)
Vx log (DM(( + NDM Buck +w) ww)9(1) Vx log 4t
IDM go-buck
VDM (c)x),
W
(13)
where w refers to the vanilla CFG scale.
D ADDITIONAL ABLATION STUDY
We present Fig. here as a supplement to Fig (w.xt. the sampling interval of IV-mixed Sampler) in the main paper. As illuatrated in Fig. 3 and Fig. 7, it can be noticed that "IV-IV" performs significantly better than "VI-IV" under almost all settings. Furthermore, we visualize Karras's 's noise schedule of our proposed dynamic CFG scale in Fig. 8 for clear understanding.
E CAN IV-mixed Sampler WORK ON CONSISTENCY MODEL?
To verify the effectiveness of IV-mixed Sampler the distillation-bann accelerated sampling model. we apply /V-mixed Sampler to the motion consistency model (MCM) (Zhai et al., 2024). We use the Ani Animatedi llioon version from the official MCM library and set the number of
19

---

Published as a conference paper at ICLR 2025
1th Frame
5th Frame
9th Frame
13th Frame
Figure 10: Visualization of our /V-mixed Sampler on Animatediff (SD V1.5, Motion Adapter V3) using SD XL as the IDM.
sampling steps to 4. After extensive empirical experimentation, we found that the sampling mech- anism shown in Fig. can enhance MCM performance by introducing IDM. Specifically, we used three checkpoints: the VDM w/o distillation, the VDM w/ distillation, and the standard IDM. During each sampling step with the VDM w/ distillation, IV-mixed sampling is performed using the VDM w/o distillation and the standard IDM. important to note that if the VDM w/ distillation used for IV-mixed sampling (L.e., replacing the VDM w/o distillation with the VDM w/ distillation), it will not work due to its LCMSchedules at each step, which is deteeeneriiii and would disrupt the semantic information provided by the IDM.
We present the results in Table 8, and we can find that /V-ixeed Sampler significantly improves the performance of MCM in all metrics.
F HOW TO MAKE IV-ixeed Sampler PERFORM BETTER ON MODELSCOPE-T2V?
The performance of /V-mixed Sampler largely depends on the generative capacity the IDM and the VDM. In the previous version of this study, we used SD V1.5 as the IDM on Modelscope- but now, we choose SealVis-V6.00 V6.0-B1 (Mage,Space, 2023), which less sensitive to resolution, as the IDM. We then adjust the CFG scales wioo of the IDM and VDM to 2 (uback (Woack -3), respectively, and present the results in Table 9. From these results, we observe that IV-mxxed Sampler using Real Vis-VisvV..00 V6.0-B1 as the IDM outperforms FreeInit across all metrics, which highlights the potential of our algorithm.
2) and
G WHY CANNOT IDM AND VDM USE DIFFERENT VAES?
In the denoising process /V-mixed Sampler, if the latent spaces of the VAEs in IDM and VDM are different, /V-mixed Sumpler will not function properly. This is similar to attempting direct image classification with a language model without fine-tuning; their nigh-dimensional semantic spaces different and therefore incompatible. We demonstrate this by applying SD XL as the IDM to Animatediff (SD) V1.5, Motion Adapter V3), as shown in Fig. 10. As illustrated Fig. 10, the resulting video frames are entirely black and cannot be recognized by the naked eye.
H HOw TO MAKE IV-mixed Sampler INDEPENDENT VAE TO REALIZE PERFORMANCE ENHANCEMENT OF MOCHI-1-PREVIENT
We made minor modifications to /V-mixed Sampler to eliminate the need for ensuring that the VAE of IDM and VDM are the same. To be specific, the changes made to Mochi-1 are shown in Fig. 11 of the revised version. focusing on converting the video latent 2, to Zg using one-step sampling, enhancing it with the ControlNet tile (SD XL), and adding certain amount of random noise before passing it through Euler's 's inversion (similar to Inversivnrs..n). This approach allows the video latent to be converted back to 2,, thereby improving video fidelity through IDM.
The visualization results can be found in Figs. 12, 13 and 14, where it can be found that IV-mixed Sampler greatly improves the video fidelity. For example, for the prompt "A serene forest clearing
20

---

Published as a conference paper at ICLR 2025
Michi-1
VAE
eimescep
Latent
Doe ZCM to
Hochs-3
(tho Jast time)
zetine
951
Mochi-1
Mochi-1
VAE Dec
Latent
Labent
Image
Irage
Video
Iateot
Flosttuier
Inversion
ciresttep 952
(three sCope)
Edeen tep 976
cinnstep
Latent
Video
add Gamezian
Image
Video
Nodae
cimea
cimeetepp
(0030 o'00y 67
Ieage
.*.
Ixage
Image
..*
Irage
you
we
Video
VAE Dec:
Labent
Mochi-1
Moohi-2
Latent
Flowbster
Plowkuler
FlowSules-
Eloestep
singie step)
Einestep
(three acepa)
timestop
976
Figure 11: The illustration IV-mixed Sampler on the state-of-the-art (SOTA) open-source VDM Mochi-
Table 9: Modelscope-T2V with Better IDM on Chronomagic-Bnnch...
Model
Method
Extm Model
(MTEPVD())
UMTScore (f)
GPT4o-MTSooe (1)
Standard DDIM
2000
2.65
2.97=0013
ModelScope-Tee T2V
FreeInit
X
2.99
3.10=mms
IVmisedd Sompler (SD V1.5)
234.90
(+. 6.7
3.00 (+ 0.35)
2.16 (+ 0.19)
WVmiied Sumpler (Rc.0VV--V6.-BB)
217.25 21.36)
3.3
3.25
0.25)
at dawn, where deer graze peacefully while golden rays of sunlight pierce through the mist-laden trees.". ", IV-mixed Sampler not only improves visual quality but also makes the deer the video walk very naturally, which is really amazing!
I WHEN WOULD /V-mixed Sampler PERFORM WORSE THAN VANILLA SAMPLING?
In the certain dimension of VBench, particularly the domain related temporal, the performance of /V-mixed Sampler is inferior to that of Vanilla Sampling. We believe that /V-mixed Sampler improves the overall quality of the composite video by balaneing visual quality and temporal co- herence. However, as shown in Fig. 15, slight inconsistencies can arise if the magnitude of motion change is too large within a few consecutive frames. The best approach to address this issue is to balance visual quality and temporal coherence based on the hyperparameter 2, as introduced in Appendix A.4.
21

---

Published as a conference paper at ICLR 2025
1st frame
1st frame
30th frame
30th frame
58th frame
58th frame
84th frame
84th frame
A serene forest clearing at dawn, where deer graze peacefully while golden rays of sunlight pierce through the mist- laden trees
Figure 12: Left: IV-mixed Sampler. Right: Vanilla Sampling.
22

---

Published as a conference paper at ICLR 2025
1st frame
1st frame
30th frame
30th frame
58th frame
58th frame
84th frame
84th frame
An explorer standing at the edge of massive desert canyon, with swirling sands below and towering rock formations stretching into the distance.
Figure 13: Left: IV-mixed Sampler. Right: Vanilla Sampling.
23

---

Published as a conference paper at ICLR 2025
1st frame
1st frame
30th frame
30th frame
58th frame
58th frame
84th frame
84th frame
A cozy mountain cabin in the midst of a snowfall, with warm light emanating from the windows and smoke curling from the chimney.
Figure 14; Left: IV-mixed Sampler. Right: Vanilla Sampling.
24

---

Published as a conference paper at ICLR 2025
1th Frame
2nd Frame
3rd Frame
4th Frame
a bear catching a salmon in its powerful jaws
Figure 15: A selected example from VBench's subject consistency dimension.
25

---

Published as a conference paper at ICLR 2025
J VISUALIZATION
In order to avoid the size of the paper being too large for the reader, we downsample the video frames and present them here. We present the synthesized video visualization of Animatediff (SD V1.5, Motion Adapter V3) in Fig. 16-17, the synthesized video visualization of ModeIScope-T2V Fig. 18 and the synthesized video visualization of VideoCrafterV2 in Fig. 19.
Sonder Sampling
f1A FEA
J
Wontnal Sumpler
Soatted Sampligg
"c444644
Wenited Surepter
Stostand Sempiig
Sampler
Stonledd Sampping
Werntel Sampler
Figure 16: The synthesized video visualization of Animatediff (SD V1.5, Motion Adapter V3),
26

---

Published as a conference paper at ICLR 2025
Sowder Sampling
Wonined Sampler
Nowstrr Samppig
Weemitad Strrpter
Stariied
[Voriood Sempler
85
Soopled Samping
Werntel Sampler
Figure 17: The synthesized video visualization of Animatediff (SD V1.5, Motion Adapter V3).
27

---

Published as a conference paper at ICLR 2025
Sooder Sampling
Nowstrri Samppig
Weriited Surrpte
Sempler Sampper
Scopdeed Sompinn
Werneel Sampler
Figure 18: The synthesized vidco visualization of ModeIScope-T2V.
28

---

Published as a conference paper at ICLR 2025
Sautet
Drease
Wessed
Stuadesd Stappey
Figure 19: The synthesized video visualization of VideoCrafterV2.
29


Reviews:
[{"review_id": "7lLaTJE2pw", "reviewer": "Reviewer_ViGW", "summary": "The paper presents a novel sampling scheme for video diffusion models that leverages pre-train image diffusion models, as they generally have higher visual fidelity compared to current open source video models. The sampling method takes steps forward / backward (DDIM inversion / DDIM) in the diffusion model, alternating between the score functions of the image and video models. Results show high fidelity visual quality while retaining consistent motion in generated video.", "questions": "* What is the variance of the quantitative metrics (e.g. FVD in Tables 1 and 2)? The values are pretty close and it’s unclear  if the results are statistically significant\n* It would be nice to have more video samples to look at (e.g. on an anonymous website). There is only one set of videos in the supplementary, and it is hard to see motion for ones in the paper appendix.\n* Does this method still work in scenarios using distilled models? E.g. 1 step or 2 step generation", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is generally clear, and well written\n* The proposed method is novel and interesting\n* Results seem to show good improvement in generation quality of the videos, while retaining consistent motion\n* Main experiments and ablations are thorough and show the benefits and tradeoffs of different instantiations of the proposed method", "weaknesses": "* The sampling process requires both I/V models to be in the same underlying latent space, which may be restrictive. How would this method also be used in cases where there is temporal downsampling in the video latent space (as this this is a very common video generation architecture)?\n* It is unclear how useful / relevant this method may be ~6+ months from now, as the main motivation of the paper is leveraging the lack of good open source video models, and public video datasets will get better (e.g. OpenVid10M [1]), and better video generation models will be released (e.g. Mochi-1 [2] as of recently).\n\n[1] https://huggingface.co/datasets/nkp37/OpenVid-1M\n[2] https://huggingface.co/genmo/mochi-1-preview", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Pemfwx7E9", "reviewer": "Reviewer_ZanA", "summary": "The paper proposes IV-Mixed Sampler, which is to combine image diffusion models (IDMs) with video diffusion models (VDMs) to get the benefit of higher image quality of IDM and good temporal consistency of VDM. This is achieved by sampling from x_t to x_{t+∆t} and from x_{t+∆t} to x_t using IDM and VDM", "questions": "See weakness", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "+ The paper combines theoretical proof with practical implementations, and is relatively well-rounded\n+ The proposed method seems to make intuitive sense", "weaknesses": "The writing of the paper can be improved. Here are several comments but not just limited to these points. Overall the presentation is a bit confusing.\n- Why do the paper keep mentioning OpenAI's strawberry in Abstract and Intro, which is closed-sourced and no paper or technical detais was released? How is this important or even related to motivating the paper?\n- Figure 2 is confusing and should not be put in the intro. This is more like ablation studies, and it is made rather confusing, especially authors also start to talk about \"R-\" in the intro out of nowhere (line 94), and the starts to discussion \"I-\" and \"V-\" in line 107 without any explanation\n- The use of \"go\", \"back\", \"begin\", \"end\" etc in the equations are also confusing\n- Plots like Fig 6 are also hard to read\n\nThe results of the model is decent but not very strong. For example, FreeInit is significantly better than ours for UMT-FVD on ModelScope-T2V", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2N6JmMMxH", "reviewer": "Reviewer_ypZs", "summary": "The paper introduces IV-Mixed Sampler, a novel algorithm designed to enhance video synthesis by leveraging the strengths of both Image Diffusion Models (IDMs) and Video Diffusion Models (VDMs). The core innovation is the use of IDMs to improve the quality of individual video frames while maintaining temporal coherence through VDMs during the sampling process. The authors claim state-of-the-art performance on four benchmarks, including UCF-101-FVD, MSR-VTT-FVD, Chronomagic-Bench-150, and Chronomagic-Bench-1649. The paper also provides a theoretical analysis of the IV-Mixed Sampler and its transformation into a standard inverse ODE process, as well as an exploration of the design space for hyperparameters.", "questions": "1. What are the detailed reasons that we can't use different VAEs for IDM and VDM when adapting the proposed method? Do you have experiment results about this?\n2. How does the IV-Mixed Sampler perform as the length and complexity of the video sequences increase? Are there any scalability issues that the authors have identified?\n3. Potential Limitations: Are there any specific scenarios or types of video content where the IV-Mixed Sampler might underperform? If so, how might these limitations be addressed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: The concept of combining IDMs and VDMs to enhance video synthesis is innovative. The paper addresses a significant gap in the field by improving the visual quality of synthesized videos while preserving temporal coherence.\n\nQuality: The paper is well-structured, with a clear problem statement, methodology, experimental validation, and conclusion. The theoretical analysis provides a solid foundation for the proposed algorithm.\n\nClarity: The authors have done an excellent job of explaining complex concepts in a clear and concise manner. The figures and tables are well-designed and aid in understanding the content.", "weaknesses": "Evaluation Metrics: The paper employs benchmarks like UCF-FVD and MSR-VTT-FVD, which can not accurately assess text-to-video generation models. More suitable benchmarks like EvalCrafter or Vbench are needed for a comprehensive evaluation.\n\nDemonstration Insufficiency: The provided demos do not clearly demonstrate the proposed method's superiority over baselines, requiring more compelling examples to showcase improvements.\n\nIllustration Clarity: Figure 3's illustrations are not clear. The images presented do not effectively convey the differences in quality and temporal coherence, making it difficult for readers to grasp the paper's points regarding the performance of various samplers. High-quality, clear visualizations are crucial for helping readers understand the nuances of video synthesis methods, and the paper would benefit from improved clarity in its visual aids.\n\nLimited Applicability: The method's compatibility with state-of-the-art video generation models, which often use distinct VAEs from image models, is limited. This restricts its potential applications and prospects in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)