FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise
Authors: Yunlong Yuan
Yuanfan Guo
Chunwei Wang
Wei Zhang
Hang Xu
Li Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: video diffusion models; Fourier transform; noise prior; frequency filtering
Primary area: generative models

Abstract: Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, as improved noise priors yield better generation results. One recent approach employs the Fourier transform to manipulate noise, marking the initial exploration of frequency operations in this context. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide a comprehensive theoretical analysis of the variance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain. Our method features a novel filtering technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distribution. Additionally, we propose a partial sampling process by perturbing the latent at an intermediate timestep while finding the noise prior, significantly reducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.

Published as a conference paper at ICLR 2025
FREQPRIOR: IMPROVING VIDEO DIFFUSION MODELS WITH FREQUENCY FILTERING GAUSSIAN NOISE
Yunlong Yuan', Yuanfan Guo?, Chunwei Wang', Wei Zhang,, Hang Xu², Li Zhang" SSchool of Data Science, Fudan University Noah's Ark Lab, Huawei
https: bttps://githube http//gggttthhhheee com/ fudan- zvg/an/gro/rrer
ABSTRACT
Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, improved noise priors yield better generation results. One recent approach employs the Fourier transform to manip- ulate noise, marking the initial exploration of frequency operations in this con- text. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide comprehensive theoretical analysis of the vari- ance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain. Our method features novel filter- ing technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distri- bution. Additionally, we propose partial sampling process by perturbing the latent at an intermediate timestep while finding the noise prior, significantly re- ducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.
INTRODUCTION
Benefiting from notable advancements of diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020: Song et al., 2021b) alongside the expansion of large video datasets (Bain al., 2021; Schuh- mann et al., 2022), text-to-video generation has experienced remarkable progress (Ho et al., 2022a; Wu al., 2022a; Blattmann al., 2023; Ge et al., 2023; Guo et al., 2024; Singer et al., 2023; Wang et al., 2023; Chen et al., 2023). In ordinary videos, the content between successive frames often shows high similarity, allowing the video to be considered as a sequence of images with motion information. Leveraging this characteristic, the architecture of video diffusion models (Blattmann et al., 2023; Wang et al., 2023; Hong et al., 2023; Guo et al., 2024) commonly incorporates tempo- ral motion layers into existing image diffusion models. In addition to model architecture, some studies, inspired by the consistent patterns observed across video frames, investigate the relation- ships within the initial noise prior. Consequently, alongside research focusing on the training and sampling phases (Song et al., 2021a; Karras et al., 2022; Lu et al., 2022; Salimans & Ho, 2022; Song et al., 2023), another important line of research in video diffusion models to explore noise initialization strategies, since improved noise prior can potentially yield better generation results.
Several efforts have been made explore the noise prior, as the initial noise significantly impacts the generated outcomes (Ge et al.. 2023; Qiu et al., 2024; Chang et al., 2024; Gu et al., 2023; Mao al., 2024; Wu al., 2024). PYoCo (Ge al., 2023) discovers that the noise maps corresponding to different frames, derived from pre-trained image diffusion model, cluster in t-SNE space (Van der Maaten Hinton, 2008), indicating strong correlation along the temporal dimension. Based on this observation, it introduces two kinds of noise prior with correlations on the frame dimension.
"Corresponding author 1 z2hangfd www.aaedu.ac.aac.ommmmm

---

Published as a conference paper at ICLR 2025
Gaussian
of 100
09
0.98
0,97
96
a² decreasing
Freelnit
1.00
0.99
0. 0.96
Ours
Figure 1: (Left) Generated video frames corresponding to Gaussian noise with different vari- ances. As the variance, denoted as o², decreases from 1.00² to 0.96², the imaging quality deterio- rates and background details are gradually lost. (Right) Comparisons of our method against the FreeInit and standard Gaussian noise. The frames generated using FreeInit appear overly smooth and blurred in the area the highlighted red box, whereas our method preserves rich image details.
However, this change in the noise prior requires massive fine-tuning. Freelnit (Wu al., 2024) investigates the low-frequency signal leakage phenomenon in the noise, as also demonstrated in the image domain (Lin et al., 2024), and finds that the denoising process is significantly influenced by the low-frequency components of initial noise. Leveraging these insights, it uses frequency filtering on the noise prior to enhance the temporal consistency of generated videos. However, despite its efforts, the generated videos suffer from excessive smoothness, limited motion dynamics, and lack of details. Moreover, additional iterations are necessary to refine the noise, with a full sampling process conducted in each iteration, making Freelnit (Wu al., 2024) quite time-consuming.
To address this gap, we conduct a mathematical analysis and provide theoretical justification. Our analysis identifies the variance decay issue existing in FreeInit (Wu et al., 2024). As depicted in Figure 1, we investigate the significance of the distribution of the initial noise for diffusion models. The impact of the variance on the quality of generated videos is evident. As decreases from 1 to 0.96, there is progressive loss of details alongside reduction in motion dynamics. The frames generated by FreeInit (Wu al., 2024) are overly smooth and lack details, as the refined noise deviates from the standard Gaussian distribution, resulting in variance decay. Therefore, it is critically important for diffusion models that the noise prior follows standard Gaussian distribution.
In this work, we introduce a novel noise prior called FreqPrior. At the core of our approach is the noise refinement stage, where we propose novel frequency filtering method designed for noise, which essentially is random variables. During this stage, retain the low-frequency signals while enriching high-frequency signals the frequency domain, thereby reducing the covariance error and ensuring that the distribution of our refined noise approximates standard Gaussian distribution. As illustrated in Figure 1, our method does not suffer from the detail loss issue present in FrecInit (Wu al., 2024). Additionally, retaining low-freque ww-frequency signals enhances semantic fidelity. Furthermore, to obtain the noise prior, we adjust the diffusion process by perturbing the latent at an intermediate step, resulting in significant time savings without compromising the quality of the generation results. We conduct extensive experiments on Vbench (Huang et al., 2024b), a comprehensive benchmark, to assess the quality of generated videos. The results demonstrate that our method effectively addresses the issue of limited dynamics while improving the overall quality. Moreover, our approach outper- forms the best on VBench, highlighting the superiority of our method. Additionally, our method achieves a time-saving of nearly 23% compared to Freelnit (Wu et al., 2024).
In summary, our contributions are as follows: (i): We propose a novel frequency filtering method designed to refine the noise, acquiring better prior, termed FregPrior. We provide a rigorous theoretical analysis of the distribution of our prior. Numerical experiments reveal the covariance error of our method is negligible, implying that prior closely approximates Gaussian distribu- tion. (ii): we propose the partial sampling strategy in our framework, which perturbs the latent at a middle timestep. It can save much time without compromising quality. (iii): Extensive experiments validate the effectiveness of FreqPrior. Specifically, our approach improves both video quality and semantic quality, achieving the highest total score over baselines VBench (Huang et al., 2024b).
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Video generative models In the field of video generation, previous work has explored range of methods, including VAEs (Kingma & Welling, 2014; Hsieh et al., 2018; Bhagat et al., 2020),
GANs (Goodfellow et al., 2014; Tian et al., 2021; Brooks et al., 2022; Skorokhodov et al., 2022), and auto-regressive models (Wu et al., 2021; 2022a; Ge al., 2022; Hong al., 2023). Recently dif- fusion models (Ho et al., 2020; Song et al., 2021b; Sohl-Dickstein et al., 2015; Dhariwal & Nichol, 2021) have showcased great abilities in image synthesis (Rombach et al., 2022; Saharia et al., 2022;
Nichol et al., 2022), and pave the way towards video generation (Ho et al., 2022b; He et al., 2022;
Voleti al., 2022). Many recent works (Ho et al., 2022a; Blattmann et al., 2023; Ge et al., 2023; Guo et al., 2024; Wang et al., 2023; Chen et al., 2023) on video synthesis are text-to-vided diffu- sion paradigm with text as highly intuitive and informative instruction. Both ModelScope (Wang et al. 2023; Luo et al., 2023) and VideoCrafter (Chen et al., 2023) are built upon on the UNet (Ron- neberger et al., 2015) architecture. VideoCrafter adds a temporal transformer after a spatial trans- former in each block, while in ModelScopoe each block comprises spatial and temporal convolution layers, along with spatial and temporal attention layers. AnimateDiff (Guo et al., 2024) generates videos by integrating Stable Diffusion (Rombach et al., 2022) with motion modules.
Noise prior for diffusion models Given inherent high correlations within video data, several stud- ies (Ge et al., 2023; Qiu et al., 2024; Chang et al., 2024; Gu et al., 2023; Mao et al., 2024; Wu et al., 2024) have delved into the realm of noise prior within diffusion models. Both FreeNoise (Qiu al., 2024) and VidRD (Gu et al., 2023) focus on initialization strategies for long video generation, with FreeNoise employing a shuffle strategy to create noise sequences with long-range relationships, while VidRD utilizes the latent feature of the initial video clip. f-noise prior interprets noise continuously integrated noise field rather than discrete pixel values (Chang ct al., 2024). However, focuses low-level features, making it more suitable for tasks such as video restoration and video editing. Mao et al. (2024) identifies that some pixel blocks of initial noise correspond to certain concepts, enabling semantic-level generation. Nevertheless, collecting these blocks for different concepts is time-consuming, which limits its practical application. Motivated by correlations in the noise maps corresponding to different frames, PYoCo (Ge al., 2023) carefully designs mixed noise prior and progressive noise prior. FreeInit (Wu et al., 2024) identifies signal leakage in the low-frequency domain and uses Fourier transform to refine the noise, making the initial exploration of frequency operations in the noise prior. However, noise is essentially different from signals, mak- ing the classic frequency filtering method unsuitable. As a result, the generated videos lack motion dynamics and imaging details due to the variance decay issue. To address these limitations, propose a novel prior to enhance the overall quality of generated videos.
METHOD
FreqPrior comprises three key stages: sampling process, diffusion process, and noise refinement, as shown in Figure 2. To obtain a new noise prior, our method starts with Gaussian noise, which then goes through these three stages sequentially, repeated several times, to result in refined noise prior. Once the new prior is established, it serves as the initial latent for video diffusion models to generate video. The sampling process in our framework is DDIM sampling (Song et al., 2021a).
3.1 DIFFUSION PROCESS
During the sampling process, the latent becomes clean. Unlike the conventional diffusion process that typically diffuses the clean latent timestep T, our approach perturbs the latent with the initial noise e once sampling reaches a specific intermediate timestep, denoted as t. The diffusion process can be formulated as follows by leveraging the Markov property:
aT a2
Enoisc
(1)
where {āz TT are the notations corresponding to the diffusion scheduler (Ho et al., 2020), and i represents the i-th iteration.
The rationale for conducting the diffusion process beforehand stems from the observation that when I reaches about timestep 400, the latent 2 has roughly taken shape and resembles the clean latent
3

---

Published as a conference paper at ICLR 2025
Sampling process
D
DM
DM
Iterations: n....
Generated video
Noise refinement
Diffusion process
NoiseRefine
Anoton
stepl: Preparation of two sets of noise
Step2: Retention of LF signals
Step3: Post-processing
while emriching III signals
FFT
Znoise
3(+..)
((12))
== MOŠ
(1 - M²) oy
2
SEFT 98(4.2)
Gasmins
-3(4
The framework of FreqPrior. It consists of three stages: sampling process, diffusion process, and and noise refinement. In the noise refinement stage, the noise is refined three steps including noise preparation, noise processing, and post-processing.
Figure 2:
z). indicating the latent already has recovered large low-frequency information. Consequently, this modification yields nearly identical outcomes compared to diffusing a pure clean latent. This mod- ification offers a notable advantage in terms of efficiency, it significantly reduces the number of required sampling steps while maintaining consistent results. Therefore, we achieve substantial time savings without compromising the fidelity of our results.
3.2 NOISE REFINEMENT
The noise refinement stage focuses on processing different frequency components of the noise to improve video generation quality. now-frequency signals help the model generate videos with bet- ter semantics, while high-frequency signals contribute to finer image details. Unlike conventional filtering methods, which typically target signals like images, our approach processes noise, essen- tially random variables, distinguishing it from traditional techniques. Therefore, we propose a novel frequency filtering method designed to effectively handle noise, enhancing overall quality.
Step 1: Preparation of two sets of noise We begin by preparing two distinct sets of noise, each serving specific purpose: one convey low-frequency information and the other to provide high- frequency information. Initially, we independently sample from a standard Gaussian distribution to obtain wi. 7/. yf and y2. where yf and y correspond to high-frequency information. As for low-frequency information, combine zi 2oose with vi and T to yield x and x² follows:
x
(cos 0 zhoise sin mi),
o N0,
VI + cos y
(2)
x
(cos 0 / + +sin 0
n
T ~ ~(0,
cos
Here, ratio cos controls the proportion of zhoise contained within xj and x It adds flexibility to the framework, allowing control the amount of low-frequency information derived from poisee

---

Published as a conference paper at ICLR 2025
Algorithm 1 FreqPrior
Require:
T: total diffusion step; t: middle timestep; {a]E=o scheduler. n: number of iterations. Initialize Zς = c, where ~ (N,
2: b Obtain the noise prior
3: for i = 0 to ni
4: 2e <- Sampling(2n ZT)
Partial sampling process
5: Inoise = v Vor/ / on Ze +
Diffusion Process
6: Zr + NoiscRefinne [noise)
> Noise refinement
7: Generate a video from new noise prior
8: 2g f Sampling(zr)
b Sampling process
9: video +- Decode(zo)
10: return video
Step 2: Retention of low-frequency signals while enriching high-frequency signals
We apply
the Fourier transform to map the noise to the frequency domain:
x = FaD(x)) X = FaD(x)), yi = FsD(y)), y = FaD
(3)
where FAD represents the Fourier transform operation on temporal and spatial dimensions. We then perform filtering with low-pass filter M:
こ M x (1 M3)0 oyi E =
+ (1- o
(4)
Since we are filtering Gaussian variables rather than real image signals, the conventional filtering approach may not be suitable. Typically, a high-pass filter is set to (1 -M), we use (1 -M 1 instead. This adjustment is inspired by a fact in probability: u, v (), are independent, then for E [0, 1]. it holds that w m u + (1 m²)0.5 v is also standard Gaussian. In traditional filtering operations, the sum of the low-pass and high-pass filters equals one. However, in approach, the sum of the squares of the low-pass and high-pass filters equals one. This modification enriches the high-frequency signals, maintaining the balance between low-frequency and high- figgurrey components. As result, it mitigates the loss of details and motion dynamics, leading to higher fidelity in the generated videos.
Step 3: Post-processing After filtering, the frequency features are mapped back into the latent space, followed by post-processing to form the new noise prior 2+1 The process as follows:
(R (2t.1) 3 (2/r)) + R (2r:2) (x),2)) x1.1123= = =ab (5)
Unlike traditional methods that overlook the imaginary component, our approach recognizes the im- portance of the information contained within these imaginary parts, which are crucial for preserving the variance the noise prior. Consequently, we retain both the real and imaginary components. Specifically, we take both the positive real parts of z and 21 but for imaginary components, we take the positive imaginary part of z and the negative imaginary part of ziy This is the reason we prepare sets of noise in Step This symmetric formulation enhances the retention of valuable information while effectively eliminating unnecessary and complex terms.
In summary, our framework comprises two phases: the first phase focuses on finding new noise prior, while the second phase generates a video based on that prior. The process of finding the noise prior includes the sampling process, diffusion process, and noise refinement, as previously discussed. Our framework is detailed in Algorithm 1.
3.3 ANALYSIS ON THE DISTRIBUTION OF DIFFERENT NOISE PRIOR
For the mixed noise prior proposed in PYoCo (Ge et al., 2023), it is constructed as follows:
1
22 Ej +
V2 Eshare: Ej, Eshore A I),
(6)
5

---

Published as a conference paper at ICLR 2025
where 2, and ç represent the j-th frame of latent z and Gaussian noise The noise prior has correlations in the frame dimension, as each frame consists of shared noise Eshored:
Cov 2,) = 0.5I, i
Therefore, considering only the frame dimension, the diagonal elements of the covariance matrix are 1, and others are 0.5, which deviates standard Gaussian distribution. Similarly, the distribution of progressive noise prior also deviates from standard Gaussian distribution.
To conduct a theoretical analysis for Freelnit (Wu et al., 2024) and our method, we first need to determine the distribution of the refined noise. We begin with the following assumption:
Assumption 1. After the diffusion process, Znoise follows standard Gaussian distribation ( I).
We focus on the frame, height, and width dimensions, as other dimensions do not affect analysis. The noise prior of FreeInit (Wu et al., 2024) has the following distribution (see Appendix B.1);
2 z~ CO.N
(AAA+ BAB), P N
where R/ is the vector form of the noise prior, N is the length of z, A is the diagonal matrix corresponding to the low-pass filter M, and A and B represent real and imaginary parts of 3D) Fourier matrix as illustrated in Appendix A.2.
Similarly, the distribution our method is as follows (see Appendix B.2):
z ~~~ N (0, --((0.r
cos² Q²)
1 (AAB + BAA).
+ cos 0
To measure the deviations of two Gaussian distributions, we introduce the concept of covariance error.
Definition 3.1 (Covariance error). For two Gaussian variables with the same expectations, (p, E1) and N(u, Σ2), the covariance error defined as the Frobenius norm of the difference between their covariance matrices: Σ ΣzllE.
Under the condition of the same low-pass filter M, we can derive the relationship of the covariance error of Freelnit and our method by using Equation 61 and Theorem C.I:
cos² 0
E Freg || I 1t cos 0 Σ FreeInerl]..
(10)
This inequality indicates that I- EEvrggttorrll II-EFreatnilll:
I-cos- 2 50%. This demonstrates that the refined noise produced by our method is closer to a standard Gaussian distribution. Our approach theoretically reduce the covariance error by at least 50% compared to FreeInit (Wu al., 2024). To further investigate the covariance error, we conduct numerical experiments with three different shapes and types of low-pass filters: the Butterworth filter and the Gaussian filter. All computa- tions are performed with float64 precision.
Table 1: Numerical experiments on covariance error. We report the covariance errors for three types of prior under various settings, including three different latent shapes and two different filters. The mixed noise prior is independent of filters.
(16, 20, 20)
(16, 30, 30)
(16, 40, 40)
Prior
Butterworth
Gaussian
Butterwonth
Gaussian
Butterworth
Gaussian
Mixed
154.9193
232 3790
309. 839.3387
FreeInit
3.8230
8,5878
5.7001
12.8817
7.6026
17.1756
8.5071 10
7.7218
10-28
1.4002 10-25
1.2656
10 26
2.7342 10 28 2.4140 10-26
As illustrated in Table our proposed noise prior exhibits the lowest covariance errors, which are minimal and can be considered negligible. Freelnit shows some covariance errors, indicating the presence of a variance decay issue. The covariance errors for the mixed noise prior are significantly higher, suggesting that it deviates substantially from standard Gaussian distribution. These nu- merical experiments imply that our noise prior can be regarded as a standard Gaussian distribution.

---

Published as a conference paper at ICLR 2025
Table Main results. For different types of noise prior, we provide the settings for finding the prior and sampling steps for video generation. Evaluation metrics include quality score, semantic score, and total score. Additionally, we report the inference time, which includes the time for finding the noise prior and the time for generation.
Base model
Noise prior
Prior finding
Generation
Quality
Semantic
Total
Inference time
Gaussian
25 steps
69.50
54.92
66.58
27.73s
Mixed
25 steps
Progressive
25 steps
VideoCrafter
Gaussian
3+25 steps
69.75
58.10
67.42
83.09s
FreeInit
sampling partial sampling
25 steps
70.62
58.97
68.29
83.
Ours
25 steps
70.63
61.33
68.77
63.67s
Gaussian
50 steps
73.13
65.69
71.64
19.24s
Mixed
50 steps
Progressive
50 steps
ModelScope
Gaussian
3+50 steps
73.25
66.31
71.87
57.72s
FreeInit
full sampling
50 steps
73.61
67.24
72.34
57.738
Ours
partial sampling
50 steps
74.04
69.06
73.04
44.88s
Gaussian
25 steps
79.56
69.03
77.45
23.34s
Mixed
25 steps
Progressive
25 steps
AnimateDiff
Gaussian
3*25 steps
79.49
69.71
77.54
70.22s
FreeInit
2 full sampling partial sampling
25 steps
79.58
68.85
77.43
70.45s
Qurs
25 steps
80.05
70.37
78.11
54.05s
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Baselines In our experiments, we establish the following baselines: Gaussian noise, mixed noise, progressive noise, and Freelnit (Wu et al., 2024). Gaussian noise serves as the default prior for diffusion models. The mixed noise prior and progressive noise prior proposed by PYoCo (Ge et al., 2023). FreeInit is the pioneering work that employs Fourier transform to create a new prior.
Implementations We conduct the experiments on three open-soruce text-to-video diffusion mod- els: VideoCrafter (Chen et al., 2023), ModelScope (Wang et al., 2023), and AnimateDifff (Guo al., 2024). DDIM (Song et al., 2021a) set to the default sampler, with the scheduler's 's offset configured 1. Both FreeInit and our method require additional samplings acquire the noise prior, with the number of extra sampling iterations set to 2. To ensure fairness, we use Butterworth Filter with normalized ppatial-tempora cutoff frequency of 0.25 as the low-pass filter for both FreeInit and our method. In our approach, the timestep is set to 321, the ratio cos is set to 0.8 for ModelScope and AnimateDiff, and 0.7 for VideoCrafter. All experiments are conducted on NVIDIA V100 GPUs. For more details, please refer to Appendix D.
Evaluation To evaluate the performance of each noise prior, we use VBench (Huang et al., 2024b), a comprehensive benchmark that closely aligns with human perception. VBench dissects evaluation into specific, hierarchical, and disentangled dimensions, each featuring tailored prompts and evalu- ation methods. Specifically, VBench assesses performance across two primary levels: quality score and semantic
The total score calculated the weighted average of the quality semantic score. The scores range from 0 to 100, with a higher score indicating better performance in the corresponding aspects. For each noise prior, we generate 4730 videos for VBench evaluation. For more details, please refer to Appendix E.
and
4.2
MAIN RESULTS
Quantitative results As shown in Table 2, our method achieves the highest scores across all met- rics, quality score, semantic score, and total score, on the three different base models, underscoring the superiority of our proposed noise prior. Our approach enhances both the video fidelity and

---

Published as a conference paper at ICLR 2025
Progresstre
Figure 3: Generation results using PYoCo prior. Both mixed noise prior and progressive noise prior lead to crashes on pretrained video diffusion models.
mantic consistency of the generated videos. In contrast, the mixed noise prior and progressive noise prior lead to crashes and failure generating normal videos, as illustrated in Figure 3. This is due to the significant gap between these types of prior and the standard Gaussian distribution, as these types of prior introduce correlations in the frame dimension. The PYoCo method requires training a model specifically these types of prior and cannot be directly applied to pre-trained diffusion models, which limits its practical applications. FreeInit and our method require two additional samplings to acquire the noise prior, resulting in increased inference time. To investigate whether the perfor- mance improvements are attributed to more denoising steps, we triple the steps during generation for Gaussian noise. While tripling the steps for Gaussian noise provides slight performance boost, the improvements are modest, particularly on ModelScope and AnimateDifff where the total score increases by only 0.23 and 0.09, respectively. Although it shows of 0.84 on VideoCrafter, its total score still falls well short of both FreeInit and our proposed prior. FreeInit generally enhances performance compared to Gaussian noise prior, it reduces the semantic score and total score on AnimateDiff. The reason may be the negative effects of variance decay surpass the positive effects of refinement on low-frequency signals. Our method does not have such a variance decay issue. Overall, when compared to Gaussian noise with triple steps and Freelnit, our method outperforms metrics while requiring the least inference time, saving approximately 23%. The performance gains stem from the noise refinement stage, where we introduce a new fre- quency filtering method targeted at the noise. The time savings arise from diffusing the latent at an intermediate step, resulting partial sampling that reduces several denoising steps.
significant improvement
Qualitative results Figure 4 presents comparative visualization of the results. In the top left case, our method produces video frames with superior fidelity, featuring backgrounds reminiscent of café, while the frames generated using Gaussian noise lack any background. FreeInit further deteriorates the result compared to Gaussian noise, blurring the area within the red box into an indistinct speck. The top right case demonstrates that the videos generated by method exhibit finer details and better semantics. In the middle left case, our results are aesthetically superior in terms of color and brightness, while those produced by Gaussian noise appear relatively dim. The middle right case highlights that both baselines fail to generate a guitar, whereas our method successfully creates one that aligns closely with the provided text prompt. the bottom left case, the example generated by FreeInit resembles "a cat sleeping in bowl" rather than "a cat eating food out of bowl." the bottom right case, the video generated from Gaussian noise is missing a "person," while FreeInit produces an unnatural representation, lacking motion dynamics. In contrast, our method delivers the highest quality video, featuring a person walking forward. Overall, these cases illustrate that our method outperforms these types of noise prior in both quality and semantics.
Table A Alation study on the impact of ratio cos O. We present total score across various values of ratio cos 0. To eliminate the effects of timestep i, it is fixed to O.
cos
VideoCrafter ModelScope
AnimateDiff
1.0
69.02
72.82
78.07
0.9
68.96
72.92
78.07
0.8
68.91
73.12
78.12
0.7
69.04
72.97
78.09
4.3 ABLATION STUDY
Influence of ratio cos 9 The generation results are affected by two byper-parametery the ratio cos 0 in the noise refinement stage and the timestep / in the diffusion process. To investigate the
8

---

Published as a conference paper at ICLR 2025
Geussion
Gaussion
Ouars
Ours
person drinking coffee in cafe
person walking in the snowstorm
shipl
Goossinn
Gaussion
Froeini
Owrs
happy fuzzy panda playing guitar nearby campfire, snow mountain in the background
boat sailing smoothly calm lake
Fresinit
Freeiwit
Ours
A cat eating food out of a bowl
person walking the snowstorm
Figure 4: Qualitative results and comparisons. The cases in the top row are generated using AnimateDiff, while the middle row displays cases from ModelScope, and the bottom row shows cases generated by VideoCrafter. For each case, present the generation results from different types of noise prior along with the corresponding prompt.

---

Published as a conference paper at ICLR 2025
Food
56
- 0
06 0
LO 0.7
cos 0
Figure 5: Generation results on different values of cos O. Though there are some changes in the generated video frames cos varies, they are quite similar.
effects of cos 0, set timestep i to 0 to eliminate the influence of t. In Equation (2), both x] and x2 contribute to low-frequency signals of 2 the initial latent for the subsequent iteration. As cos 9 decreases, the proportion of Anoise in xj and x diminishes, indicating a reduction in the low- frequency components rooted Znojse. cos governs the extent to which low- wwwfrequency signals retained, assuming the filter remains constant. Therefore, cos O can not be small. We conducted experiments with four different values of cos As shown in Table 3, for AnimateDiff (Guo et al., 2024) and ModelScope (Wang al., 2023), Total Score initially increases, reaching its peak at cos f = 0.8, before declining. For VideoCrafter (Chen et al., 2023), Total Score get the highest f 0.7. Overall, the differences among different cos 0 values minor, indicating the FreqPrior is robust and not sensitive to changes in cos O. The visualization results presented in Figure demonstrate that while varying cos leads some differences in the video frames, they are still quite similar.
VideoCrafter
ModelScope
AnimateDiff
75.2
Tokal Scose
72
Total Scorts
70.0
68.0
Geundinn
22.00
Gaurlien Hostrll.. Ours
Geuacan
Our
07.5
33.6
06.1
77.4
Umastop
timestag 2
timastop
Figure 6: Ablation study on the impact of timestep t. Total Score is assessed accross different diffusion timesteps i for three distinct text-to-video diffusion models. Overall, the timestep has little effect on the evaluated metric.
Influence of timestep t. Figure 6 shows that our method consistently outperforms Gaussian noise prior and FreeInit (Wu et al., 2024) across varying timestep t. While there are some fluctuations, the method all three base models shows slow declining trend, indicating that as the timestep increases, the quality of generated videos is likely to decrease. However, with larger timestep. fewer denoising steps are required in each sampling iteration to find the noise prior. Consequently, it presents a trade-off between the generation quality and the inference time. Considering both factors, we selected 321 for the final setting.
curve corresponding
CONCLUSION
We introduce a new noise prior for text-to-video diffusion models, named FreqPrior. The key stage in our framework lies noise refinement, where we propose novel frequency filtering method specifically designed for Gaussian noise. By refining the noise, we obtain a better prior for video diffusion models, thereby enhancing the quality of generation results. Although other types noise prior have been proposed to improve the performance of video diffusion models, their distributions often deviate from a standard Gaussian distribution, leading to sub-optimal generation outcomes. In practice, the covariance error between our prior and Gaussian noise approaches zero, indicating that our noise prior closely approximates a Gaussian distribution. Extensive experiments demonstrate the superiority of method over existing noise priors.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was supported in part by National Natural Science Foundation of China (Grant No. 62376060).
REFERENCES
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
Sarthak Bhagat, Shagun Uppal, Zhuyun Yin, and Nengli Lim. Disentangling multiple features in video sequences using gaussian processes in variational autoencoders. In ECCV, 2020.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
Tim Brooks, Janne Hellsten, Miik Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming- Yu Liu, Alexei A. Efros, and Tero Karras. Generating long videos of dynamic scenes. In NeurlPS, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. How i warped your noise: comporally-correlate noise prior for diffusion models. ICLR, 2024.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffuuion models for high-quality video generation. arXiv preprint, 2023.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In NewrlPS, 2021.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic VQGAN and time-sensitive transformer. In ECCV, 2022.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In ICCV, 2023.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeuriPS, 2014.
Jiaxi Gu, Shicong Wang, Haoyu Zhao, Tianyi Lu, Xing Zhang, Zuxuan Wu, Songcen Xu, Wei Zhang, Yu-Gang Jiang, and Hang Xu. Reuse and diffuse: Iterative denoising for text-to-video generation. arXiv preprint, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. In ICLR, 2024.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruigi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Im- agen video: High definition video generation with diffusion models. arXiv preprint, 2022a.

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J.
Fleet. Video diffusion models. NeurlPS, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. In ICLR, 2023.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In NewrIPS, 2018.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In NeurlPS, 2023.
Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. In ICLR, 2024a.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In CVPR, 2024b.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurIPS, 2022.
Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In ICCV, 2021.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
LAION-AL aesthetic-predictor, 2022.
URL
LAION AI/
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, 2023a.
Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In CVPR, 2023b.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan LI, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint, 2023.
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. The lottery ticket hypothesis in denoising: Towards semantic-driven initialization. In ECCV, 2024.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. In ICLR, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2025
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. CVPR, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICMI,,
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ila Sutskever. Consistency models. In ICML, 2023.
Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In ICLR, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Vikram Voleti, Alexia Milcoeur-Martinen.t and Chris Pal. MCVD masked conditional video diffusion for prediction, generation, and interpolation. In NeurIPS, 2022.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Mod- elscope toxieoo technical report. arXiv preprint, 2023.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2024.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. GODIVA: generating open-domain videos from natural descriptions. arXiv preprint, 2021.
Chenfei WU, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022a.
13

---

Published as a conference paper at ICLR 2025
Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint, 2022b.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initial- ization gap in video diffusion models. In ECCV, 2024.
A PRELIMINARY
A.1 DIFFUSION MODELS
Diffusion models (Ho al., 2020) are class of generative models that recover the data corrupted by the Gaussian noise through learning a reverse diffusion process. iteratively denoises from Gaussian noise, which corresponds learning the reverse process of fixed Markov Chain of length
T. The diffusion process Markov chain that gradually corrupts the data with Gaussian noise. For the diffusion process given the variance schedule Br:
( 2o) = II (()))==1),,
g(xe(1r-1) = Nzx v1 1111-11811).
(11)
Using the Markov property, we can sample o, at an arbitrary time from zo in closed form. Let = - 3r and Cr = II: Cts, we have
(< N N(xs Vārzo, (1 T.)
(12)
By the Bayes' rules, 9( [2, mo) can be expressed as follows:
(11-1/00, 20) = Nx= far o0), 3r1),
(13)
where jM vo) V vo + var(1 a:-1)
1 ōt-1 Br-
1 0z Dt. and
Br
(14)
1 ar
For the reverse process, generates zo from ør with prior = N 0, and transitions:
pe( N (2 1), (x 1)).
(15)
In the equation, θ are learnable parameters of models which are trained to minimize the variant of the variational bound Em,.n ||e co (2t, t))
A.2 FOURIER TRANSFORM
Discrete Fourier Transform (DFT) is of the most important discrete transforms used in digital signal processing including image processing. The discrete Fourier transform can be expressed as the DFT matrix, denoted as F, defined as follows:
0-0
w
0-1 N
-1(N-1)
1.
I11
N-
F = (w(m-1)(n-21)
N
(16)
NxN
(N-1)-0
(N
1)-1
(N-1)
W
w
where WN 2wr/NN is primitive N-th root of unity. The inverse transform, denoted F-1 be derived from F as its complex conjugate transpose, scaled by + F
can
The DFT matrix F can be decomposed into its real and imaginary parts, represented respectively by matrices A and B:
F A Bi, A=R(F) B 3
(17)
This decomposition simplifies the understanding of the structure and properties of the DFT matrix, providing deeper insights. Using Euler's formula, and B can be explicitly expressed as:
(cos ((m 1) (n 1) Nx)
B (sin ((m 1) (n 1)0)), NxN:
(18)
where 0 N. Notably, both A and B are real symmetric matrices.
14

---

Published as a conference paper at ICLR 2025
Lemma A.1. For 8 2x where N is positive integer, it holds that k= sin (k. - 1)6) for any integer I.
Proof. By applying Euler's formula, we rewrite sin (1 (k: 1)0) as 3 (wN)
where WN
2xi// Then we have:
N-1 Σ
N
Σ sin (i
3 (wNN = 3
Σ
wh
(19)
1=0
The term E
k=U whth is the sum of geometric sequence. If w = 1, then E wk = N, yielding
1k & === WN O.
Otherwise, if w'y * 1, we have Σn EN'O with = O, and consequently
(1 (1- (1-wW) (1 w/y). Since w 1, then
WN
Ik= w
In conclusion, we have shown that Σ N Lkm1
sin (1 (k 1) 0) = 0 for any integer I.
This lemma offers foundational insights into the behavior of the sum of sinusoidal functions, Now, introduce theorem regarding properties of the DFT matrix.
Theorem A.2. Given DFT matrix F CNNN with A and B representing its real and imaginary parts respectively, it holds that AB = BA = 0 and A + B = NI.
Proof. Using the property of the inverse Fourier transform, we have
I= FF (A+ Bi) (A Bi)
N (A² B² 22 -B--B +BA))
(20)
Comparing real parts and imaginary parts of both sides, we derive:
A² + B² NI, BA = AB.
(2
Considering the matrix AB, we calculate the value of the element in the m-th row and n-th column:
N
(AB)
Σ
cos ((m-1))))))) - (k: 1) sin ((k - 1) (n 1)
k=1
(22)
Σ (sin ((m + --2kk- 1)0)
sin ((m n) 1)6) 0.
2 k=1
The last equation holds using Lemma A.1. The equation holds for each element of AB. Therefore
AB BA = 0.
For the 3D Fourier transform, it can be represented as follows using the Kronecker product:
FaD = Fr @ Fn @ Fw
(23)
The inverse transform is given by:
1
F N FiD-
(24)
Similarly, we decompose Fan into its real part Aan and imaginary part B3n:
FSD (Ar Br & (A# + B (Aw Bwi),
Asp = Ar 3 Aи X Aw Ar @ B 2 Bw By & A Bw Br 2 Bh & Aw, (25)
BaD AT 3 Ah Bw + Ar & BH & Aw Br @ Ah Aw By & BH & Bw.
By Theorem A.2 and the property or Kronecker product, it still holds that:
Ag + B Ny BADAAD = Aap = 0.
(26)
It reveals that the 3D DFT matrix shares the same properties as the ordinary DFT matrix. For convenience, we denote the DFT matrix, including multi-dimensional cases F, with size denoted as N. Employing mathematical induction, we can extend Theorem A.2 from one- one-dimensional case arbitrary finite dimensions:
Theorem A.3. Given DFT matrix or multi-dimensae DFT matrix F E ON with and B its real part and imaginary part respectively, it holds that AB BA 0 and A B NI.
15

---

Published as a conference paper at ICLR 2025
B NOISE DISTRIBUTION ANALYSIS
B.1
FREEINIT
FreeInit (Wu et al., 2024) uses conventional frequency filtering methods to manipulate noise, which is the key step in the framework. This step can be formulated as follows:
zT R (Fan) (FaD (Znoise) M F3D (17 o (1 (1)
(27)
where Fan is the Fourier transform applied to both spatial and temporal dimensions. M is spatial- temporal low-pass filter. Znoise is noisy latent derived from corrupting the clean latent with initial Gaussian noise to timestep T, while is another Gaussian noise. For analysis, we focus solely on the spatial and temporal dimensions, ignoring the batchsize and channel dimensions. Additionally, we flatten the latent Zy E IR xxxx into vector Z E fho, The equation 27 can be expressed in matrix form as follows:
=
Fy,
(28)
where is the DFT matrix of transform FaD. x and y are random vectors corresponding to Znoise and 7), and Ar and Ay diagonal matrices associated with low-pass filter M and high-pass filter - MM. Therefore it holds that Ag + Av I. This equation can be simplified to the following form using equation 17:
1 z (AA,A+ BA, (AA,A+ BA,
1
(29)
Under the Assumption 1, x vec(Znoise) .V (0, I), where and y are independent. Since is a linear combination of independent Gaussian random vectors, follows that z is also Gaussian. To derive the distribution of only need to compute its expectation and covariance. The expectation is straightforward and given by [E[z] = 0. The covariance of z can be calculated as follows:
1 NZ (AA,A+ B) + NZ (A BA, B)"
Cov (z)
[A
(30)
To simplify the expression, denote P N (AA, A BA, B). Then the term can be expressed using P:
A(I A, B(I A AA, BA, B
(31)
2 B (AA, BA, NI NP.
The last equation follows from 2 NI. stated in Theorem A.3. Combining Equation (30)
and Equation (31), the covariance of z is given by:
Cov (z) p2 ((-
(32)
Consequently, we obtain the distribution of as follows:
n~ N~~ (o.
(33)
Due to the property of the low-pass filter M where each element lies between to 1, both A, and Ag semi-definite diagonal matrices. Consequently, semi-positive definite matrices. The covariance structure resembles a² + (1 a)? which is less than for a e (0, 1). This indicates difference between the distribution of z and the standard Gaussian distribution. We explore this further in Appendix C.
prove that both P and I P
B.2 FREOPRIOR
The noise refinement stage of our method consists of three distinct steps, which are elaborated on in Section 3.2. To facilitate further analysis, we express these steps matrix form. The first step, noise preparation step, can be represented as:
1
X1
(cos 0 sin 0 m)) X2 +I+ c (cos 0 sin 0 172,, (34)
+ 2 8
where 7h, 772 ~ N (0, and are independent. Under Assumption 1, x ~ NN I). Obviously, x, m,, and 1/2 are independent. Both x; and X2 linear combinations of independent of Gaussian
16

---

Published as a conference paper at ICLR 2025
random vectors. Their expectation be computed directly: E/x1] E X 0. Next,
calculate the covariance these variables. Specifically, the covariances are given by:
Cov (xi) = Cov (x2) 1 cos² 0 Cxv(x1, = Cov(x2, x1) cos 0
cos I. (35)
This implies that x1 and X2 are correlated, as they both share a component x when cos # O. The noise processing and post- post-processing steps can be expressed as follows:
F A,
F F
Ag Fyi:
22 F A + F FAA,, Fy2.
(36)
2
( + 3 (2)) v2
(zz)),
(37)
where y2, N (0, I) independent. Regarding the filters, A₂ and Ag diagonal matrices corresponding to the low-pas. tw--pass filter M and the high-pass filter (1 -M).5
The refined noise z can be expressed in following form using equation 17:
V2N (AA, BA, B + AA, B BA, 1)x1
+ (AA, + BAg B + AA, BAg A)
(38)
(AA,A+ BA, AA, BA, X2
+ (AA,A+ BA, - B
A A)
From the mathematical form of this expression, is evident that the matrices preceding these random vectors share similar structures. To simplify this equation, we introduce the following notations:
Let:
C = AA,A BA, D. = AA, BA, = AA, BA D,, AA, BA
(39)
Since A and B are real symmetric matrices, and A, and Ag are diagonal matrices, it is straight- forward to prove that C₂ and Cy are symmetric matrices, while D2 and D, are skew-symmetric matrices. Using these notations, Equation (38) can be simplified follow:
V2N - z (C2 + D.) + (C, + Dz) + (C, -C,- -.) + Dg)
(40)
In the analysis of /2N z where z a Gaussian Gaussian-distributee vector, we need to calculate the ex- pectation and covariance to determine its distribution. The expectation is given by E[z] 0. The covariance can be expressed as the sum of several covariance terms related to X1, X2. Y1 and y2- Specifically, the covariance of /2N z can be expressed as follows:
Cov(V2N: z)
Cov ((C₂ + D2) x1) + Cov ((C, Dz) x2)
+ Cov ((C,, + D2) y1) + Cov ((C, - D,) D2)
(41)
Cov ((C₂ + Dz) X1, (C₂ D2)
+ Cov ((C2 - D.) X2, (C2 + x1).
The covariance of /2N z consists of 6 terms, with the first four terms representing the covariance of each random vector. The last two terms are not independent. By solving these terms, We can derive the covariance of z.
terms that arise due the fact that x1 and x₂
First, we focus the covariance terms related y1 and y2:
Cov (IC, + Dz) + Cov (CC, y2)
= (C., + Dy) Cov (C,) + D,) + (C, - -D,) Cov (y1) (V))(C, (C,
(42)
(C,, + Dy) (C, DD) (C, D,) (C, + Dy) = 2 (C - -
Similarly, we can infer Cov ((C, + D2) x1) Cov ((C, D2)) X2) combined with Equation (35):
Cov ((C, + D2) x1) Cov (CC, D2) x²) cos2 (C² D2)
(43)
17

---

Published as a conference paper at ICLR 2025
Having computed the first four terms, now turn attention to the last two terms. With Equation (35), we have:
Cov ((C₂ + Dz)x1, (D, + Cov ((C2 -000--D00 D x2, (C2 + Dz)
- =CC, + D2) Cov(X1 (C D2) + (C2 -D)) Cov(x2, (C2 + +2)
(44)
c
cos² 0
2cos f
1 cos² (Ca (Cn+ Dx)² + cos2 (C, Dn)² + f (C² D
Substituting the expression of the covariance related X1, X2, y1 and y2 with Equa- tions (42, 43, 44), we can express the covariance of v2N in the following form:
4cos² 0
Cov (V2N x) 2 2 D C D
1 cos2 D
(45)
To further simplify this equation, we need to explore the properties of Cz. Cy, D2 and D,, From Theorem A.3, which establish AB BA 0 and A² B² NI. We can compute the squares of matrices C2 and Dr as follows:
C = AA BA, BAA,
(46)
-D- AN, A+ A
Notice that the squares of C. and D2 share a similar form, differing only in the middle matrix: one 2 and the other is B². This observation inspires have established A B = NI. Therefore, we can express it as follows:
calculate C² D especially since
C D = A (42 + B²) A, A+ BA, (B² + A2) A
(47)
= A + BBA
Since C, and Dy follow the same pattern with only the subscript replaced, it also holds that:
C D NAA A + NBA
(48)
Make use of Av A2) we can conclude:
c D C DØ NA A A A+NB + + B N N (49)
Substituting with Equations (46) and (49), we can simplifies Equation (45) to express the covariance of /2N z as follows:
Cov (V2N 2) 2N cos B'
(50)
+ cos 0
Inspired by the form of AA, A, and BA, B which are the matrix multiplication of AA, B and A. We creatively construct new matrix Q N (AA, + BA, prove Q symmetric matrix. The square of Q is follows:
BA,AA, A). It is easy
1 Q² = NZ (AA. BA A
(51)
By combining Equation (50) and Equation (51) and eliminating the constant V2N from both sides of the equation, we can calculate the covariance of z:
2 cos2 0
Cov (z) = I
+ cos 0 Q²
(52)
Finally, we derive the distribution of z as follows:
(0,
2 cos2 0
z~~ N 0,
+ cos2 Q2
(53)
It is clear that the covariance of our refined noise is "smaller" than I. However, AB BA and the diagonal elements of A, ranges from 0 to 1, gives the intuition that Q is close to We make further analysis in Appendix C.
18

---

Published as a conference paper at ICLR 2025
C COVARIANCE ERROR ANALYSIS
Theorem C.1. Given two semi-positive definite matrices C and D satisfying C' = D 0, then
|| 2 | where || || Frobenius Norm.
Proof.
Since ||C
DIE 2 0, then expanding yields:
|||| |P| tr (C D'C) 2 (CD).
(54)
The last equation holds because the and D are symmetric matrices and tr(-) is invariant under circular shifts. Then we can conclude:
| | 2 2tr (CD) 2 DII 2tr ((C D) D).
(55)
Using Cholesky decomposition, for semi-definite matrix D, there exists matrix such that D LL Then derive:
tr ((C D) = (rC () tr (C- D)-
(56)
From the given condition C = D. thus C-D O, thus L D) L semi sosiivvsi Therefore the trace this matrix will non-negative. Therefore ||[|| 2 D||..
From Equations (33) and (53), we know the covariance of refined noise for each method:
PreeInit = P (I P)2, ErrePPrio = 2 cos 2
(57)
1 + cos2 0
Consider the same settings, including that the low-pass filters are identical, meaning Ag is fixed. Since the filter A₂ is diagonal with its diagonal elements in the range [0, 1), we have 0 I A, I. Consequently, we obtain the following inequality for matrix P:
0=p= N (AA,A+ BA 1 N
(A² +²= =I
(58)
Now consider the difference between Freetwit and I:
I Dreelnit (P P2) - 0.
(59)
This inequality holds because 0 < P < I, which implies p2 J P. This demonstrates that Erreefnit is indeed "smaller" than To conduct further analysis Ereelnut and Σ Ereginit we first establish the relationship between P and Q. Noticing that P and Q have similar forms, we can derive the following results by leveraging these specific forms:
p2 Q² N2 ((AA, Ba A, + (AA,BAA, BA,
(60)
1
B = N BA =P.
Combining Equations (59) and (60), we obtain:
cos2 0
cos2
cos
I- E FregPrior
1 + cos2 0 Q² z
1 cos2 (P-
cos2 (I E Freeelna). (61)
Then we can analyze the covariance errors (as defined in Definition 3.1) by applying Theorem C.I:
cos2 0
I Σ FregPPiorll
< 1 cos? | 7- EFrecfnitlp:
(62)
In practice, for common continuous low-pass filters, such as Butterworth filters and Gaussian filters, the corresponding function values monotonically decrease as the frequency increases. Given that AB BA and Q = (AAB + BAA), it follows that Q² intuitively close to a zero matrix, making the covariance error nearly zero. This is further corroborated by numerical experiments.
19

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL DETAILS
Three open-sourced text-to-video models are used as the base models for evaluation: They AnimateDiff (Guo et al., 2024), ModelScope (Wang al., 2023; Luo al., 2023), and VideoCrafter (Chen et al., 2023).
For Diff we use 1or---dv155.22 motion module along with realistic Vision v20_v20 dreambooth LoRA! sampling 16 frames of at resolution of 512 x 512 at 8 FPS with guidance scale of 7.5.
For ModelScope, we utilize the godellsope-damo---x----vvooooooo pling 16 frames at a resolution of 256 x 256 at 8 FPS, with a guidance scale of 9.
version, sam-
For VideoCrafter, we employ the VideoCrafter-vl base text-to-video model, sampling 16 frames at resolution of 320 x 320 at 10 FPS, with guidance scale of 12.
E EVALUATION METRICS
We employ VBench (Huang et al., 2024b) for evaluation, a comprehensive benchmark designed with tailored prompts and evaluation dimensions specifically aimed at assessing video generation performance. A key feature of VBench is its incorporation of human preference annotations, ensur- ing alignment with human perception. VBench uses a hierarchical and disentangled scoring system, breaking the overall total score into two main components: quality score and semantic score. It covers 16 evaluation dimensions, with contributing to quality score and 9 contributing to seman- tic score. Each dimension assessed using specially designed approach, ensuring precise and meaningful evaluation of the generated videos. The assessments involve various off-the-shelf mod- els (Caron et al., 2021; Ruiz al., 2023; Radford et al., 2021; Li et al., 2023b; Teed & Deng, 2020; LAION-AI, 2022; Ke et al., 2021; Wu et al.. 2022b; Li et al., 2023a; Huang et al., 2023; 2024a; Wang et al., 2024), and the score for each dimension is normalized on 100 scale, based on empirical minimum and maximum values.
Quality score is calculated as the weighted average of seven dimensions: subject consis- tency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality, and imaging quality. The weight for aesthetic quality set to 2, while the other dimensions carry weight of 1.
semantic score is calculated as the weighted average of nine dimensions: object class, mul- tiple objects, human action, color, spatial relationship, scene, spenaaapeeaance style, temporal style, and overall consistency, with each dimension equally weighted 1.
After calculating quality score and semantic score, total score is calculated as follows:
Total
Wy Quality
-emantic
(63)
Wy + Ws
Wi + w.
where w. and w. are 4 and 1 respectively by default.
F VISUALIZATION RESULTS
More qualitative results. Additional qualitative results are presented in Figure 8, The videos generated using our noise prior exhibit superior video quality, in terms of imaging details, aesthetic aspects, and semantic coherence.
Visualization of the influence of timestep t. As illustrated in Figure 7, the first three rows of frames, which correspond to different timesteps I, are almost the same. In the fourth case, there are some differences in the representation of the grape stem, highlighted by red box. The stem absent at the timestep of 321. At the timestep of 321, the stem is missing. The final case demonstrates more notable differences; as the timestep increases, the ice cream appears to melt, and the changes are observable on the table. The visualizations suggest two key points: first, in most instances, the
wtt.r//atmmtttcccocoomooooooooom v20.safctensors
20

---

Published as a conference paper at ICLR 2025
1=1
I=
1=
1-
1-
F- 401
1= 401
2- -1
C- 161
2-241
321
001
1-1
241
6-1
c- B1
T-1
T-241
C-1
401
Visualization results the influence of timestep. We present five cases where other settings are fixed to isolate the effects of varying timestep t. Overall, timestep has minimal impact on the generation outcomes. However, it does exert some influences on the imaging details occa- sionally. For the fourth and fifth cases, the red boxes highlight the differences.
Figure 7:
timestep é has minimal impact on the overall generation results; second, although the content and layout of the video frames remain largely unchanged, the timestep indeed influence the finer imaging details. In general, the differences are quite minor, which means we can save much time by diffusing the latent at intermediate timestep during the noise refinement stage without compromising the quality of generation results.
G LIMITATIONS
While our method enhances consistency and smoothness in videos generated from Gaussian noise, can occasionally result in unnatural smoothness that does not align with the laws of physics. Additionally, although our approach improves overall performance, it may alter the content layout of video frames compared to Gaussian noise. For real images, low-frequency signals typically dictate layouts; however, this is not always true for the noise prior in diffusion models. Our method refines the Gaussian noise prior using novel frequency filtering technique, which usually preserves the structural similarity to the original Gaussian noise. Nonetheless, in some cases, the generated videos can differ significantly. During filtering, high-freeuency components from other Gaussian noise may subtly change the structure of the Gaussian noise prior, resulting in variations in the content and layouts of the generated videos.
H BROADER IMPACTS
This work aims to propose novel prior by refining initial Gaussian noise to enhance the quality of video generation. Text-to-video diffusion models hold the potential revolutionize media creation and usage. While these models offer vast creative opportunities, it is crucial to address the risks of misinformation and harmful content. Before deploying these models in practice, it is essential to thoroughly investigate their design, intended applications, safety aspects, associated risks, and potential biases.
21

---

Published as a conference paper at ICLR 2025
Fresiric
Freehit
Churs
An epic tornado attacking above glowing city at night, the torado made of smoke
A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo
Goussion
Gaussion
OuAS
Ours
An oil painting of couple in formal evening wear going get caught heavy downpour with umbrellas
A cat wearing sunglasses at pool
Gaussion
Gaussian
Fresiit
Freeivit
Ours
volcano
person playing guitar
Figure 8: More qualitative results.
22


Reviews:
[{"review_id": "UfuXbP6wJA", "reviewer": "Reviewer_kmCy", "summary": "The paper introduces FreqPrior, a novel noise initialization strategy for text-to-video diffusion models.  FreqPrior refines noise in the frequency domain using a new filtering technique that addresses different frequency signals while maintaining a noise prior distribution close to a standard Gaussian distribution. This method helps preserve important low-frequency signals, enhancing semantic fidelity. The authors propose a partial sampling process that perturbs the latent space at an intermediate timestep during the noise prior generation. This approach significantly reduces inference time without compromising the quality of the generated videos. The paper provides a comprehensive theoretical analysis of the variance decay issue in existing methods, which contributes to the loss of details and motion dynamics. The authors show that the covariance error of their method is negligible, indicating that their noise prior closely approximates a Gaussian distribution.", "questions": "The title should explicitly mention \"Video Diffusion Models\" to clarify that the method is specifically designed for video generation and not applicable to image diffusion models. This will avoid any confusion and make the scope of the paper clearer to readers.\n\nThe paper should provide detailed measurements of GPU memory usage before and after applying the proposed FreqPrior method, particularly focusing on peak memory consumption. Given that 3D FFT can be memory-intensive, especially for resolutions higher than 512x512, this information is crucial for understanding the practical feasibility of the method. Include tables or graphs showing the GPU memory usage for different resolutions and compare them with the baseline methods. This will help readers assess the trade-offs between memory consumption and performance improvements.\n\nThe paper should explore the impact of different Classifier-Free Guidance (CFG) strengths when using FreqPrior. Since CFG is a common technique used in diffusion models to enhance generation quality, understanding how FreqPrior interacts with varying CFG strengths is important for practical applications.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main contributions are:\n\nFreqPrior refines noise in the frequency domain using a new filtering technique that addresses different frequency signals while maintaining a noise prior distribution close to a standard Gaussian distribution. This method helps preserve important low-frequency signals, enhancing semantic fidelity.\n\n The authors propose a partial sampling process that perturbs the latent space at an intermediate timestep during the noise prior generation. This approach significantly reduces inference time without compromising the quality of the generated videos.\n\nThe paper provides a comprehensive theoretical analysis of the variance decay issue in existing methods, which contributes to the loss of details and motion dynamics. The authors show that the covariance error of their method is negligible, indicating that their noise prior closely approximates a Gaussian distribution.", "weaknesses": "The title should explicitly mention \"Video Diffusion Models\" to clarify that the method is specifically designed for video generation and not applicable to image diffusion models. This will avoid any confusion and make the scope of the paper clearer to readers.\n\nThe paper should provide detailed measurements of GPU memory usage before and after applying the proposed FreqPrior method, particularly focusing on peak memory consumption. Given that 3D FFT can be memory-intensive, especially for resolutions higher than 512x512, this information is crucial for understanding the practical feasibility of the method. Include tables or graphs showing the GPU memory usage for different resolutions and compare them with the baseline methods. This will help readers assess the trade-offs between memory consumption and performance improvements.\n\n\nThe paper should explore the impact of different Classifier-Free Guidance (CFG) strengths when using FreqPrior. Since CFG is a common technique used in diffusion models to enhance generation quality, understanding how FreqPrior interacts with varying CFG strengths is important for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ieDZTquPtE", "reviewer": "Reviewer_PCyZ", "summary": "The paper presents a novel approach for enhancing noise priors in text-to-video diffusion models. The authors introduce a new frequency filtering method to refine noise priors, improving video quality by preserving essential details and dynamics better than existing baselines such as Gaussian noise, mixed noise, progressive noise, and FreeInit. The core motivation is to keep the standard Gaussian distribution for the frequency-based sampling refinement process. The method requires additional sampling iterations but offers notable performance improvements across multiple metrics evaluated on the VBench benchmark. The experiments are conducted using three open-source text-to-video models (VideoCrafter, ModelScope, and AnimateDiff), and the results highlight that the proposed method outperforms the baselines in both quantitative and qualitative aspects.", "questions": "1. My first question is the experimental analysis, why only Vbench is provided?\n\n2. The second question is theoretical evidence for why non-normalized Gaussian distribution will cause the worse motion dynamics.\n\n3. Have you considered or tested other types of frequency filtering (e.g., adaptive filtering methods) to optimize the noise prior? What is the generalization capability of such frequency filtering? It would be important to demonstrate their broader applicability\n\n4. Have you measured the standard deviation for your generated videos with different seeds? It contains lots of randomness in video generation. Whether this work select videos based on human visualization? If not, it takes which principles for results selection?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work identifies the importance of standard Gaussian distribution in the sampling process for video generation. \n\n2. This work introduces a new frequency decomposition strategy for random variables.\n\n3. Extensive experiments and theoretical derivation provide a great illustration for the motivation.", "weaknesses": "1. Despite this work has shown the side effects of non-uniformed sampling noise distribution, it is still hard to understand why this will destroy the motion dynamics from the theoretical perspectives. \n\n2. The evaluation of this work is only based on VBench, which is somehow not sufficient. It is suggested to include more comparisons in terms of FID, FVD, etc. Whether the conclusion will stand under these metrics.\n\n3. This work lacks user study and does not provide the detailed prompts used for video generation. Since the video quality measurement for AIGCs is not absolutely reliable, providing a user study for video generation is essential. \n\n4. How to obtain the equation (7), it needs a detailed explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vt2IOcnbUG", "reviewer": "Reviewer_QxjB", "summary": "To address the problem of variance-decreasing in FreeInit, the authors propose to re-design the low-pass filter in FreeInit and use two sets of noise to maintain the variance of intermediate diffusion variables. Experiments show that the proposed method is able to preserve more details than FreeInit.", "questions": "What is the performance on more recent ODE-based diffusion models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The FreqPrior approach addresses detail loss and motion dynamics issues better than previous methods (e.g., FreeInit), leading to improved video fidelity.\n2.  The partial sampling process significantly reduces inference time by around 23% compared to similar methods.\n3.  FreqPrior achieves higher scores in quality and semantics in evaluations, especially on the VBench benchmark.", "weaknesses": "The paper argues that the variance-decreasing problem in FreeIniit causes it to generate over-smoothed results. But the provided evidence is weak. Although the demo cases at the beginning of this paper support this conclusion, more videos in the supplement files do not verify it.  According to Table 2,  the quantitative improvements over FreeIniit is also marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C71eNytuJ0", "reviewer": "Reviewer_PvRQ", "summary": "Building on FreeInit, this method introduces a novel frequency filtering approach to obtain an improved noise prior that enhances high-frequency signals and approximates a Gaussian distribution, refining text-to-video diffusion models. \nAdditionally, by implementing partial sampling instead of the full sampling used in FreeInit, it effectively reduces the sampling time.", "questions": "Questions are listed in the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Comprehensive theoretical analysis of the variance decay issue of existing methods and addressing the issue by novel filtering technique is interesting and novel.\n\n2. Extensive experiments validate the novel filtering method refine the text-to-video diffusion models significantly.", "weaknesses": "1. This work builds upon FreeInit, so the implementation of FreeInit should remain consistent with the original. However, while the original FreeInit uses 4 extra iterations, the comparisons in this work are made with only 2 extra iterations.\n\n* What would the results be if both FreeInit and FreqPrior were implemented with 4 extra iterations? Would FreqPrior still outperform FreeInit?\n\n2. Applying this method to recent T2V models could enhance the completeness of the paper.\n* If high-quality T2V models are available, making low-frequency matching unnecessary, would this method still be effective?\n* Additionally, if possible, could the method demonstrate effectiveness on the latest T2V models, such as T2V-Turbo or Pyramidal Flow?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)