How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models
Authors: Pascal Chang
Jingwei Tang
Markus Gross
Vinicius C. Azevedo
Venue: ICLR 2024 oral (2024)
Keywords: diffusion models; temporal coherency; Gaussian noise field; continuous white noise; noise transport
Primary area: generative models
TL;DR: We propose a method to warp a Gaussian noise sample while keeping it Gaussian and apply it to diffusion models to help temporal coherency.

Abstract: Video editing and generation methods often rely on pre-trained image-based diffusion models. During the diffusion process, however, the reliance on rudimentary noise sampling techniques that do not preserve correlations present in subsequent frames of a video is detrimental to the quality of the results. This either produces high-frequency flickering, or texture-sticking artifacts that are not amenable to post-processing. With this in mind, we propose a novel method for preserving temporal correlations in a sequence of noise samples. This approach is materialized by a novel noise representation, dubbed $\int$-noise (integral noise), that reinterprets individual noise samples as a continuously integrated noise field: pixel values do not represent discrete values, but are rather the integral of an underlying infinite-resolution noise over the pixel area. Additionally, we propose a carefully tailored transport method that uses $\int$-noise to accurately advect noise samples over a sequence of frames, maximizing the correlation between different frames while also preserving the noise properties. Our results demonstrate that the proposed $\int$-noise can be used for a variety of tasks, such as video restoration, surrogate rendering, and conditional video generation.

Published as a conference paper at ICLR 2024
HOW I WARPED YOUR NOISE: A TEMPORALLY- CORRELATED NOISE PRIOR FOR DIFFUSION MODELS
Pascal Chang, Jingwei Tang, Markus Gross, Vinicius C. Azevedo
<firstname>, <lastname> e.nf.eth..ch. ETH Zürich, Switzerland
ABSTRACT
Video editing and generation methods often rely on pre-trained amage-baser dif- fusion models. During the diffusion process, however, the reliance rudimen- tary noise sampling techniques that do not preserve correlations present in sub- sequent frames of video is detrimental ther produces high-frequency flickering, or texture-sticking artifacts that are not amenable post-processing. With this in mind, we propose novel method for preserving temporal correlations in sequence of noise samples. This approach is materialized by a novel noise representation, dubbed f-noise (integral noise), that reinterprets individual noise samples as a continuously integrated noise field: pixel values do not represent discrete values, but are rather the integral of an underlying infinite-resolution noise over the pixel area. Additionally, we pro- pose a carefully tailored transport method that uses f-nois to accurately ad- vect noise samples over sequence of frames, maximizing the correlation be- tween different frames while also preserving the noise properties. Our results demonstrate that the proposed f-noise be used for variety of tasks, such as video restoration, surrogate rendering, and conditional video generation. See wwt.iieee.tttttte github. io for video results.
the quality of the results. This ei-
INTRODUCTION
Despite their notable ability to generate high-quality images (Rombach et al. 2022 Nichol et al. 2022] from simple text inputs, diffusion models (Sohl-Dickstein et al.) 2015 Ho et al. 2020 have been applied with limited success when it comes to video processing and generation. While it is a common practice to build diffusion- based video pipelines upon pretrained text-to-image models
(Ceylan et al. 2023 Geyer et al. 2023 Yang et al. al. 2203 Khachatryan et al.) 2023), these approaches do not have a systematic way to preserve the natural correlations between subsequent frames from video. One contributing factor to this problem is the indiscriminate use of noise corrupt data during inference: all relevant motion information present in a video is lost in the diffusion process. Employing different set of noises for each frame within video sequences yields results that manifest. high-freyuency hallucinations that are challenging to eliminate, and some approaches resort to fine-
tuning neural networks with temporal attention layers (Liew al.] 2023 Liu et al. 2023b Shin
et al. 2023 Zhao et al., 2023 Wu et al. 2022).
Other than using random sets of noise at inference, it also common employ fixed noise as a content-agnostic solution for artificially enforcing correlations. However, the problem of fixed noise signals manifests as texture-sticking artifacts that are oblivious to post-process techniques. Alternatively, one can also invert the set of noises that reconstruct given image sequence under a conditional prompt (Ceylan et al., 2023 Geyer et al. 2023). The limitation of such an approach is that the set of inverted noises closely entangles the temporal information with the content of the frames, making such methods not suitable for injecting the learned temporal correlation into arbi- trary noise samples. Lastly, techniques such as feature warping (Ni al. 2023 Yang et al. 2023), or cross-frret frame attention (Ceylan et al., 2023 Geyer et al. 2023 Yang et al. 2023 Khachatryan also alleviate temporal coherency issues to certain extent. However, these ap-
et al.) 2023) proaches are limited because the features are not able to represent high frequencies patterns of the fine resolution image. Given the importance of the role of the noise in inferring diffusion models,

---

Published as a conference paper at ICLR 2024
Input
Random Noise
Fixed Noise
Warped Noise (Ours)
(a) Realistic render w/ SDEdit
(b) Pose-to-perse videos w/ PIDM
Figure 1: Our noise warping method lifts diffusion- image editing methods like SDEdit (Meng et al. 2022) and Person Image Diffusion Model (PIDM) (Bhunia et al., (022) to the temporal do- main. avoids unnatural flickering and texture sticking artifacts (see colored squares) that com- monly appears with standard noise priors.
it's surprising how little research was devoted exploring the impact of noise priors temporal coherency. Thus, we were motivated by the simple question: how can we create a noise prior that preserves correlations present in image sequence?
Our answer to warp a noise signal, such that it generates a new noise sample that preserves the correlations induced motion vectors or optical flow. However, implementing such approach is not so simple. The commonly employed Gaussian noise has special properties that are lost when warped/transported with standard methods. Transport relies the sub-sampling of noise sig- nal in an undeformed space. These sampling operations create important cross-correlations that detrimental to the preservation of the noise power spectral density and the independence be- tween pixels within a single sample. Our paper addresses the outlined limitations in transporting noise fields with three novel contributions. The first one is reinterpret individual noise samples used in diffusion models as continuously integrated noise field: pixel values do not represent discrete values, but are rather the integral of an underlying infinite noise. We dub this new terpretation of discrete noise sample as fnoise (integral noise). The second contribution is the derivation of the noise transport equation, which allows accurate, of a continuously defined noise signal. Lastly, we design carefully tailored transport algorithm that discretizes the noise transport equation, maximizing the temporal correlation between samples while also preserving its original properties. Our results demonstrate that our noise warping method creates temporally-correr correlated noise fields, which can be used for variety tasks such as video restoration and editing, surrogate rendering, and conditional video generation.
prepttrvnn--eeeriingg transport
2

---

Published as a conference paper at ICLR 2024
2 METHOD
2.1 THE (-NOISE REPRESENTATION
A discrete 2D Gaussian noise of dimension D x D is represented by the func- tion G (i E fl..... D12 - X1, that maps a pixel coordinate (i. to random variable j. Diffusion models employ this discrete formulation, and random variables are assumed to be independently and identically dis- tributed (i.i.d)) Gaussian samples Xt, (0, 1). At the core of our work the reinterpretation of this discrete 2D Gaussian noise as the integral of an underlying infinite noise field.
An infinite-resolution noise field is represented by a 2D white Gaussian noise signal. To construct that, start by endowing the domain E [0, Dj [0, DJ with the usual Borel o-algebra 3 = ( and the standard Lebesgue measure v. Then, the white Gaussian noise the o-finite measure space (F, E, v) is defined as a function W E 3 - (A) ~ N ((A) that maps - subset of the domain
variable, with variance ((A) (Walsh] 2006). There are many valid ways to subdivide the domain representing the continuous noise. The standard discrete setting partitions the domain E into D x D regularly spaced non-overlapping square subsets. We denote this partition as A C 3 (level 0 in the inset image). Another way of partitioning the domain is to further refine E into a higher resolution set A C & (levels l 1, 2, ..... in the inset image, where each k-th level below subdivides each pixel in level into N₁ 2 x 2 sub-pixels.) Due to the properties of white Gaussian noise, integrating sub-pixels of the noise defined on A maintains the properties of noise defined A If we assume there is only one single pixel sample in the domain (D = 1) A = [0, 1] x [0. 1]. with Ak (AK, A representing the N₂ sub-pixels finer resolution k, the following holds:
No ((A
No A = (A9).
The proposed -noise refers to the idea that instead of representing noise value in discrete point of domain, we rather represent the white noise integral over a pre-specified area. This property is represented in Equation (I) and more thorough explanation is presented in Appendix [B.] Since we assume that each pixel on the coarsest level A has unit area, the noise variance 2/2 = m(A))
each level is implicitly scaled by the sub-pixel area M4 1/Nx. While impossible to sample the noise A 0 in the infinite setting, we show in Section 2.2 that approximating it with a righhroresolution grid sufficient for temporally coherent noise transport.
Conditional white noise sampling. In practice, after obtaining an priori noise, c.g. from noise inversion techniques in diffusion models, one important aspect is how to construct the -nnoie at A from samples defined at A This is fundamentally a conditional probability question: given the value of an entire pixel, what is the distribution of its sub-pixels values? Let W W(A)) (W+))..... ..., W ()
(N0 vad) be the Ni-dimensional Gaussian random variable rep- resenting sub-pixels of a single pixel. Then, the conditional distribution (A+)) = x) is
(W(A+) x) ~ N (F.. E), with ç
u, N%
I N₂
IN.
No un mm") (2)
where u (I,...,1)T By setting = V Σ, the reparameterization trick gives us a simple way to sample W W(AA) ((Aa))ss)))))) as
(W(A))((A'))
UZ
I N₁ VNx
(Z (Z)u), with Z ~~ (O,
(3)
where (Z) is the mean of Z. Intuitively, in order to conditionally sample the white noise under a pixel of value level k one can 1) unconditionally sample discrete N₁- 2 x 2 Gaussian sample, 2) remove its mean from it, and 3) add the pixel value x (up to a scaling factor). The full derivation of Equations (2) and (3) and the corresponding python code are included in Appendix B
3

---

Published as a conference paper at ICLR 2024
Varying the milee
Levell
Triengulete waep
Cunditional upaumpling lnvel --
0.00
0.25
0.50
0.75
1.00
Varsingg brvvll
-2
Frume
Frame T
0.00
0.25
0.50
0.75
1.00
(a) Discrete noise transport pipeline.
(b) -nose intuition in 1-D.
Figure 2: (a) The discrete noise transport equation pipeline. A subdivided pixel contour (top right) is triangulated and traced backwards from frame T frame 0 (top left). Then the warped triangulated shape is rasterized into a higher resolution approximation of the white noise (bottom). The sub-pixel values are added together, and properly scaled by Equation (5) (b) 1-D toy example. A pixel slides between two existing pixels whose values Zu, are sampled from a Gaussian dis- tribution. Bilinear interpolation creates sample of lower variance (straight line), whereas -noise would create samples that follow Brownian bridge between and 21, maintaining unit variance.
2.2
TEMPORALLY-CORREL..
DISTRIBUTION
NOISE TRANSPORT
In this section, we first introduce our proposed noise transport equation, which offers theoretical way of warping a continuously defined white noise while preserving its characteristics. Then, we present our practical implementation and show that it still retains many theoretical guarantees from the infinite setting. Lastly, we provide simple I-D example to support the analysis how the proposed f -noise balances between satisfying seemingly opposite objectives such as preserving the correlation imposed by interpolation and maintaining the original noise distribution.
White noise transport. We first assume that the noise is transported with diffeomorphic defor- mation field T E E. This mapping could be represented by an optical flow field between two frames, deformation field for image editing. Our goal is transport continuous white noise W with T a distribution- distribution-presevving manner. The resulting noise T(W) can be expressed as an Itō integral through our noise transport equation for any subset A C E as
(A))
J VT (T-(T-(())) W(T W(T ( (x)) dx,
where V is the determinant of the Jacobian of T. Intuitively, Equation (4) warps a non-empty set of the domain with the deformation field T-1, fetching values from the original white noise at the warped domain coordinates. The determinant the Jacobian necessary to rescale the samples according to the amount of local stretching that the deformation induces, while also accounting for the variance change required by the white noise definition. A detailed derivation of Equation (4) be found in the Appendix C.2 In practice, optical flow maps be non-diffeomorphic due discontinuities and disocclusions. Appendix C.4 expeaiis our treatment these cases.
Discrete Warping. The noise transport equation (Equation (4) cannot be solved in practice due the infinite nature of the white noise. Thus, we first compute the higher-resolution discrete -noiee WA**,, possibly from an a priori sample (Equation (3)) Since the pixel area can undergo a non- linear warping deformation, we subdivide its contour into smaller segments which are mapped backwards via the reverse deformation field. The warped segments define polygonal shape that then triangulated and rasterized over the high-resolution domain A* Lastly, the sub-pixels cov- ered by the warped polygon are summed together and normalized, which yields the discrete noise transport for the noise pixel at position p
G(p) = VlRp
Σ Wa(AA))
(5)
ACEO,
4

---

Published as a conference paper at ICLR 2024
Randow (6, f)
16 (p)
interp
Noian (Oung)
6G G
Fixed interp.
J Nooss (Ours)
Cross-oneariano:: Cow(GGy.
Covariance: Cov(Gr,
Figure 3: We visualize the correlation between two 4 x 4 noise samples with the warping being horizontal shift by Ar 3.6 pixels. Our -oios prior preserves the correlation between the two noise samples as well as bilinear interpolation (left), while avoiding self-correlation between pixels the warped noise (right).
where W; V W is the white noise scaled unit variance level k. and 9p A contains all subpixels at level that are covered by the warped pixel polygon, with 1S2p representing the cardinality of the set. A detailed algorithm is outlined in Appendix C.4 and illustrated in Figure 2a Note that the discrete implementation will still preserve independence between neighboring pixels in the warped result. This is because the warped polygons still form a partition of the space, so each sub-pixel in A will only belong to a single warped polygon. More details on the discretization of Equation (4) into Equation (5) can be found in Appendix C.3
Toy example in 1-D. We will demonstrate the properties of our f -noise sampling in simpler one dimensional setting. Consider 1-D set of i.i.d. random variables indexed by I {0..... n} with values represented by {fo, #i.......... En ~ (00 1), and mapping function that translates the a, where i E I and [0, Using a simple linear interpolation to compute the transported values 2; yields
discrete locations by constant TDD (i)
2g Q + (1 (1-a)z,, 2, NO0 of), with o = a1) a)
The equation above means that the variance of z; is a quadratic function a such that o² = 1 for (0, 1) and ag < 1 for a (0, 1). This shows that the linear interpolation does not preserve the original distribution of the input variables.
However, if we obtain I;-1 and v; from the integral the original distribution can be preserved. By employing the f-noise, the pixel's value that is sam- pled between f;-1 and x4 is no longer deterministic. Figure 26 shows the value of for differ- ent higher-resolution samples. Mathematically. the value of is now a Brownian bridge between neighboring æ values. In Appendix C.5 distribution given by
underlying high-resolution white noise,
we show that the value of 2 is a conditional probability
2 2|2 (poos with { Pix -
a)): -1-11100 + (1- 1 oz
(6)
Thus, our continuous noise warping can be interpreted performing stochastic process centered around the result of linear interpolation. This stochastic component precisely compensates the diminished variance induced by the linear interpolation, resulting in unit variance for any a. This the intuition behind why our f -noise warping method is able to preserve the distribution of the noise sample after warping.
Long-term temporal coherency. The method outlined by previous sections only explains how consistently warp a noise sample with single deformation field. To apply this idea to a full sequence of frames, there are two possible solutions. The first one is to warp the noise frame by However, frame, such that the noise G, of the n-th frame computed G, = -n noises computed at each frame are no longer coherent with each other in this use the accumulated deformation field warp back the first sample,
the
case. A better solution where the high resolution representation is always the same, i.e., Gn = To (Go). This produces noises that are more coherent over long periods of time.
EXPERIMENTS AND RESULTS
3.1 VALIDATING JNOSSE PRIOR
Our proposed noise prior simultaneously achieves two seemingly competing objectives: maximizing the correct correlation between the warped and the original sample, and maintaining the indepen-
5

---

Published as a conference paper at ICLR 2024
Random Noise
Fixed Noise
Warped noise (Ours)
(a) 4x super-resolution
(b) JPEG compression restoration
Figure 4: Qualitative comparison of our noise warping method with baselines for video restoration
tasks using image models by SB consecutive frames comparison (top), x-t slice (bottom).
dence of pixels within each sample. These two aspects can be visualized using cross-covariance and covariance matrices of two noise samples Go and G₁ respectively. As shown in Figure 3 our -noise prior achieves the same noise cross-correlation as the bilinear interpolation, while concur- rently maintaining the same level of independence between pixels in the second noise sample G1 as methods such as resampling ("random") ") or reusing the previous noise ("fixed").
3.2 APPLICATIONS IN DIFFUSION-BASEI TASKS
We validate our noise prior on four tasks: realistic appearance transfer with SDEdit (Meng et al, 2022): video restoration and super-resolution with I'SB (Liu et al. 2023a); pose-to-person video generation (Bhunia et al. 2022). and fluid simulation super-resolution. While we show qualitative examples of these applications in this paper, we urge the reader to check out the supplementary videos on our project webpage for better visual comparisons.
Baselines. In many of our applications, we lift models trained images to work videos. Without any particular treatment, the default Random Noise prior uses different and independent noise sam- ples for each frame. Alternatively, many approaches employ a Fixed Noise prior to reduce flickering artifacts, which reuses a same fixed set of noise samples for all frames. We compare the proposed -noise mainly against these two baselines, but more comparisons can be found in Appendix D
Photorealistic appearance transfer for videos. We lift the stroke-based editing capabilities of SDEdit (Meng al. 2022) the temporal domain. We showcase results on the LSUN Bedroom dataset (Yu et al. 2015) by replacing the stroke paintings with simple renders of a synthetic bedroom scene to ensure that the inputs are temporally coherent. SDEdit then corrupts these frames and denoise them into realistic images. Figure la shows that our fnoise prior significantly improves the temporal coherency of the denoised sequence by avoiding unnatural sticking artifacts. While method also works without it, we use cross-frame attention in this specific application to better showcase the difference between the different noise priors.
Video restoration and super-resolution. In I'SB, Liu et al. (2023)) use Schrödinger bridges to diffuse between two distributions without going through the normal Gaussian distribution. The authors show examples for JPEG compression restoration and 4x image super-resolution. When using their model on a framframe basis for videos, the resulting images suffer from artifacts due to the choice of noise. By leveraging the motion information in the corrupted sequence to warp the

---

Published as a conference paper at ICLR 2024
Random noise
Fixed noise
f-noie (ours)
Figure 5: Fluid 4x super-resolution As shown in x-t slices (bottom row), Random Noise creates incoherent details (noise in the slice) while Fixed Noise suffers from sticking artifacts (vertical lines in the slice). Our f -noise moves the fluid in smoother way.
noise, we can noticeably reduce these temporal aberrations, shown in the x-t plots of Figure 4 Quantitative comparisons in Table 1 substantiate these observations.
Pose-to-person video. Person Image Diffusion Model (PIDM) (Bhunia et al. 2022 is non-latent diffusion- baased method that takes an image of a person and a pose, and generates an image of the same person with the input pose. When applying the model in frame-by-frame fashion on a sequence of poses, it either creates substantial amount of unnatural flickering when using the Random Noise prior, or produces obvious cloth -taturrn artifacts when combined with fixed noise (see blue squares in Figure lb). We estimate a rough motion of the entire body from the pose sequence and show that this information can be utilized with our noise warping method alleviate the texture sticking issue. Figure Tb shows how fnnois translates the cloth textures along with the person's movements more naturally. Interestingly, this does not fully agree with the quantitative evaluation, and we further analyse this discrepancy in Appendix D.3
Fluid simulation super-resolution. Finally, we apply our /noise prior to a challenging task of fluid super-resolution. Fluid simulations contains large motions and deformations which makes artifacts from standard noise priors much more visible. Large deformations also means that simpler warping methods like bilinear interpolation will fail miserably. We train non-latent unconditional diffusion model from scratch on a 2D fluid simulation dataset generated with the fluid solver from Tang al. 22021). The z-- plots in Figure show show that warping the noise along with the fluid density creates smoother transitions between frames.
Quantitative evaluation. We quantitatively evaluate noise prior against the baselines presented above, as well as some traditional interpolation methods in Table (T. We further add the noise priors from Ge al. (2023) and Chen et al. (2023) our evaluation. Following Geyer et al. (2023): Ceylan et al. (2023; E Lai et al. (2018). we use the warping error as metric for temporal coherency. Realism and visual quality of the results are assessed with FID Heusel et al. (2017). Improved Precision Kyn Kynkäänniemi et al. (2019) and/or LPIPS Zhang et al. (2018) whenever it makes sense. Additionally, we report the runtime of each method in Appendix C.7 C.7
Our method is computationally less efficient than the baselines in Table D but remains comparable DDIM inversion.
Method
Appearance Transfer
Video SR (4x)
JPRG restore
Pose-to-Peree -ertonn
warp (x
FID 1
Precision f
warp J
LPIPS 1
warp
LPIPS
warp
Random
10.00
74.75
0.719
8.91
0.192
8.28
0.163
34.22
Fixed
4.35
93.56
0.644
7.97
0.179
7.54
0.163
2.26
PYoCo (mixed)
7.26
81.60
0.674
8.48
0.190
7.77
0.166
20.92
PYoCo (prog.)
4.90
84.63
0.667
8.10
0.190
7.48
0.163
11.97
Control-A- A-Vien
5.09
90.82
0.649
7.98
0.192
7.73
0.164
6.45
Bilinear
3.15
143.24
0.201
21.47
0.59n
5.26
0.431
2.13 0
Bicubic
4.95
149.85
0.212
13.02
0.490
5.74
0.372
2.40
Nearest
15.10
154.73
0.344
14.30
0.329
7.91
0.213
9.21
[-noise (ours) 2.50
92.63
0.661
6.49
0.196
5.98
0.165
2.92
Table 1: Quantitative evaluation of our method on the different applications. Our method outper- forms existing noise priors in terms of temporal coherency, while being competitive to traditional interpolation methods. Additionally, our noise prior is able to generate results with visual quality on par with standard Gaussian priors thanks to its distribution-preservee properties.
7

---

Published as a conference paper at ICLR 2024
Bilinear interpolation
Bicubic interpolation
Nearest neighbor
f -noise (ours)
Figure 6:
We compare our f-noiee against standard interpolation methods on the task of photore- alistic appearance transfer for videos. For each method, we visualize a frame from the denoised result, a patch from one of the noise we used, the covariance matrix of the noise and the overall noise distribution. The covariance matrix is computed on batch of 100 noises on a small 4 x 4 patch inside the visualized noise. The standard normal distribution is plotted in orange.
4 ABLATION STUDIES AND DISCUSSIONS
Standard interpolation vs f-noise. Figure shows that bicubic and bilinear interpolation schemes for noise warping lead to blurry results, because the warped noise has less variance (histogram insets). While nearest neighbor interpolation preserves the distribution to a certain extent, it contains duplicating artifacts in regions of stretching (noise inset), which induces self-correlation between pixels. Our proposed method is able to retain high-frequency details in the result, and ensures complete independence between the pixels of the noise samples.
The noise upsampling factor k. The main parameter of our noise warping algorithm is the choice of the upsample factor k. A smaller factor k means the warped polygons are rasterized on larger sub- pixels, increasing the chances that the polygon covers none of the sub-pixels. In this case, the resulting pixel value after warping is undefined. We test our method on the fluid example, where large deformations present. The inset plot shows the ratio of unde- fined pixels in the warped noise for different values of and differ- ent frames. The further the frame, the more undefined pixels appear due to the extreme warping the pixel polygons go through. Overall, sub-sampling the noise to level k: = already largely reduces the amount of unrasterized pixel polygons. We found setting A is sufficient for most cases. For very long sequences, we devise a resampling strategy detailed in Appendix CC.4
Ratted untudted phode Ex)
Upaampling lest
Noise warping in latent diffusion models. Our experiments have shown that noise warping has limited impact on temporal coherency in latent diffusion models (LDM) (Rombach et al. 2022). This is because the noise used in latent diffusion models has lower resolution, controlling mostly the compositional aspects and low frequency structures of the generated image. This not only limits the amount of motion that can be transferred to it but also offloads the responsibility of generating the final high-frequency details to the decoder part of the image autoencoder. In Appendix H. we provide an extensive set of tests that highlight the impact of using the warped noise LDMs.
RELATED WORK
Diffusion models (Sohl-Dickstein al. 2015 Ho al. 2020)) represent the state the art for gen- erative models, surpassing previous techniques in terms of sampling quality and mode coverage (Dhariwal & Nichol 202T Xiao al. 2022). Diffusion models train neural network to reverse diffusion process, representing the data manifold through Gaussian distribution. The set of sam- pled noises, along with integrator parameters, can be interpreted the latent space of generative et al. 2023). Since their original conception, diffusion plethora tasks such as text-to-image (Rombach et al. 2022 Nichol et al. 2022 Ramesh al. 2022 Balaji et al. 2022); text-to-video (Ho al.) 2022a b Singer et al. 2022; Villegas et al. 2022). image-to peee-to-image Parmar et al. (2023) and image-to- video
diffusion models Hlluberman-Spiegell
models have been employed
Ni et al. (2023) generation; image (Meng et al. 2022 Valevski et al.] 2022) and video (Yang al.
2023
Chen et al. 2023, Ceylan et al. (023) editing; and image (Saharia et al. 2022 Lugmayr et al.
2022
Chung et 2022) and video impainting Liew et al. (2023).
Diffusion-based image editing through noise inversion. It is common practice for image editing
methods (Tumanyan al. 2022 Ceylan et al. 2023 Mokady et al. 2022 Parmar al. 2023
8

---

Published as a conference paper at ICLR 2024
Huberman Hluberman-Spiegelgiaae et al., 2023 Wu & De la Torr 2022: Yang et al.) 2023 Geyer et al. 2023 to invert the denoising process, recovering the noise that reconstructs an image under a textual condition. The inverted noise partially encodes the image structural composition, and through its recombination with a new textual prompt during the denoising process, edited images can be eff- ciently generated. This inversion process can either reconstruct a single (DDIM) (Hertz al. 2022 Mokady et al.] 2022] [Song et al. (202]) or multiple (DDPM) (Huberman -Siegnnnl--ieglllas et al. 2023
Wu & De la Torre (022) noise samples per image. Inverting the noise can be inaccurate and ex- act inversion can be obtained by coupled transformations (Wallace et al., 2022) or by bi-direction integration approximation (Zhang et al. 2023). To further increase spatial fidelity to the original image when implementing prompt-to-prompt edits, some approaches advocate for injecting cross- frame attention (Hertz et al. 2022 Parmar et al. 2023) and features maps (Tumanyan et al.) 2022) from the source image during the denoising process. Noise inversion techniques were also used to perform image-to-image style transfer (Zhang et al. 2022, Ruta et al.) 2023), combined with a noise regularization technique (Parmar et al. 2023) and improved by pivotal tuning and null-text
optimization (Mokady et al. 2022).
Disfusion-based video generation and editing. Similar to image editing, many video editing ap- plications rely on a text-to-image models to invert the noise for each individual frame. Temporal coherency is central challenge for these applications. Injecting cross-attention features (Ceylan let al. 2023; Geyer et al., [0233] Yang et al., 2023] Khachatryan et al. (023) from an anchor frame possible solution to ensure consistent edits. Recent techniques rely on modifying the noise or
the latent space of diffusion models to achieve better temporal coherency: Ni et al. (2023) warp the
latent features from a diffusion model of image-to-video pipeline: Blattmann et al. (2023) align
the latent codes of a pre-trained image diffusion model during fine-tuning; Khachatryan et al. (2023)
apply pre-specified translation vectors to the inverted noise from an anchor frame: Ge et al. (023)
construct correlated noise samples to train a neural network for generating videos from text: Chen
jet al. (2023) use a residual noise sampling that spatially preserve noise samples representing
image
pixels not moving in the video sequence. Alternatively, Layered Neural Atlases (Kasten et al.
2021
decomposes the vidco into a canonical editing space (Couairon et al. 2023 Chai et al. 2023
Image and video editing through fine-tuning, An alternative approach to deal with issues arising from image and video editing is to fine-tune neural networks on top of diffusion models. Control-net (Zhag & Agrawala 2023) learns task-specific conditions (depth and normal maps, edges, poses) that are useful for preserving structures while editing images. For editing videos, several approaches fine-tune neural network with temporal-attention modules (Liew et al.) 2023 Liu et al.) 2003bb Shii
et al. 2023 Zhao et al. 2023 Wu et al. (222) to ensure temporally coherent outputs. Alternatively, embedding (Chu et al. 2023) or disentangling (Guo al. (023) motion and the content from videos is also an effective approach to control consistency.
LIMITATIONS & CONCLUSION
In this paper, we proposed a novel temporally-correltedd noise prior. Our f f-noise representation reinterprets noise samples as the integral of higher resolution white noise field, allowing variance- preserving sampling operations that are able to continuously interpolate a noise field. Moreover, we derived a novel noise transport equation, which accounts not only for the deformation of the underlying pixel shapes but also for the necessary variance rescaling. The proposed method creates cemporally-correlated and distribution-preservinn noise samples that are useful for a variety of tasks.
Our method comes with some limitations. While more accurate, the proposed noise warping al- gorithm is computationally more inefficient than simpler techniques (Ge et al.) 2023 Chen et al.) 2023) for correlation preservation. Moreover, the underlying assumption we made is that a more cemporally-coreelan
noise prior would result better temporal coherency in video diffusion tasks. This assumption is not always guaranteed. The degree to which this holds depends on the training data and the pipeline of the diffusion model: while methods such as SDEdit give enough freedom to the noise for it to have visible impact the denoised result, more constrained pipelines can be much more oblivious to the choice of the noise prior. For future work, we believe that our noise warping algorithm can be further used to train neural networks that generate video inputs. Extending the noise prior to latent diffusion models can be another interesting direction. And that, dear reader, is how warped your noise.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
1. For applications using existing methods, use the checkpoints provided by the authors of the methods,
2. Proofs are provided in Appendix B and
3. Pseudocode are provided in Appendix B.3 and C.4
4. Extra experimental results are provided in Appendix D
5. Detailed analysis of our method in latent diffusion models is provided in Appendix
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. eDiff-I: Texa-to-Image Diffusion Models with an Ensemble Expert Denoisers. 2022. URL
http://arxiv.. org/ wbs/abs//////ooo..../..../a..aaa 221122211111 01324
Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laakso- nen, Mubarak Shah, and Fahad Shahbaz Khan. Person Image Synthesis via Denoising Diffusion Model. nov 2022. URL arxiv. org/ abs /ttp:/22....11011
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High- High-Resolution Video Synthesis with Latent Diffusion Models. JEEE Conference Computer Vision and Pattern Recognition (CVPR) 2023, pp. 27-29, 2023. (RLpttt:://aait wwtpsiiiiirrrroooooooooo abs 2304 08818
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy Mitra. Pix2Video: Video Editing using Image Diffusion. 2023.
org abs /20/2/233333 2303. 12688
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. StableVideo: dexen Consistency- aware Diffusion Video Editing. 2023.
09592
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-A- Video: Controllable Text-to-Vide Generation with Diffusion Models. 2023. URL htttp::/arxlv.org/bbb/////////
13840
Emnie Chu, Tzuhsuan Huang. Shuo- Lin, and Jun-Cheng Chen. McDM: Mediating Image Dif- fusion Models for Video-to-Video Translation with Temporal Correspondence Guidance. 2023.
URL http:h/arxxxxii http://xxxxvv.og/aab//// wwtporgggabbaaaaaa 10079
Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving Diffusion Models for Inverse Problems using Manifold Constraints. (NcurIPS):1-28 2022. ISSN 10495258. URL
http: org/ abs 2200 00941
Paul Couairon, Clément Rambour, Jean-EEmanuel Haugeard, and Nicolas Thome. VidEdit: Zero- Shot and Spatially Aware Text-Driven Video Editing. pp. 1-17, 2023. URL http://arxiv..
org/abs/ 2306 08707
Keenan Crane. Digital geometry processing with discrete exterior calculus. In ACM SIGGRAPHI 2013 Courses, SIGGRAPH 2013, SIGGRAPH '13, New York, NY, USA, 2013. ACM. ISBN 9781450323390. doi: 10. 145/2504443..5044422
ROBERT C Dalang and Thomas S Mountford. Level sets, bubbles and excursions of a Brownian sheet. Infinite Dimensional Stochastic Analysis (Amsterdam, 1999), Verh. Afd. Natuurkd, 1:117- 128, 1999.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs Image Synthesis. Advances in Neural Information Processing Systems, 11:8780-8794, may 2021. ISSN 10495258. URL
http:://arae org/ abs 2105 05233
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-V Yu Liu, and Yogesh Balaji. Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models. 2023. http://pp://axi..oorr//e///e/e/p /230 10474
10

---

Published as a conference paper at ICLR 2024
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. TokenFlow: Consistent Diffusion Fea- 2023. /
tures for Consistent Video
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7297-7306, 2018.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. AnimateDiff: Animate Your Personalized to Image Diffusion Models without Specific Tuning. 2023. URL
http://arxiv org/ abs / 2307 04725
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-Prompt Image Editing with Cross Attention Control. pp. 1-19, 2022. URL http:
//arxiv.orq abs 2208. 01626
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, (e0o-Decem(NeurIPS 2020):1-25, 2020. ISSN 10495258.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P.
Kingma, Ben Poole, Mohammad Norouzi, David Fleet, and Tim Salimans. Imagen Video: High Definition Video Generation with Diffusion Models. oct 2022a. doi: 10448550/arixx221000000. URL
/2100. 02303
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models. apr 2022b. ISSN 10495258. doi: 104485500arxxx22204444555
URL
wwwpxivv.orgaaeo abs /204 03458
William Holt and Duy Nguyen. Introduction to Bayesian Data Imputation. SSRN Electronic Jour- nal, 2023. ISSN 1556-5068, doi: 10.2139/ssm.4443333 URL htt.ttss://w...stt www.wwww....n..../ooo com/
abstract=44990
Inbar Hlueerman-Spiegelgaaa Vladimir Kulikov, and Tomer Michaeli. An Edit Friendly DDPM Noise Space: Inversion and Manipulations. 2023. URL httt:://axxivoorg/abbb...
06140
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions on Graphics, 40(6), sep 2021. ISSN 15577368. doi:
10.1145/3478511.34480
URL http:://axiv...ro. wttp///..///////........ 11418
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-Zero: Image Diffusion Mod- els are Zero-Shot Video Generators. 2023. Utt/http://rxxx..co
abs// 2303. 13439
Tuomas Kynkäääniemi,, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Wei-Sheng Lai, Jia-Bin Huang. Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming- Hsuan Yang. Learning blind video temporal consistency. In Proceedings of the European conference com- puter vision (ECCV), pp. 170-185, 2018.
Jun Hao Liew, Hanshu Yan, Jianfeng Zhang. Zhongcong Xu, and Jiashi Feng. MagicEdit: High- Fidelity and Temporally Coherent Video Editing. pp. 1-8, 2023. URL http::a/atxiv..oot//
abs/2308. 14e.147495 4749
Guan-Homg Liu, Arash Vahdat, De-An Huang. Evangelos A. Theodorou. Weili Nie, and Anima Anandkumar. 12SB: Immaee Schrödinger Bridge. feb 2023a. doi: 10.48550/arxiv.2302. 05872. URL htt.::ttt//aaa
org. wttp:vv.rrga/bb//////beee 05872

---

Published as a conference paper at ICLR 2024
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-P2P: Video Editing with Cross-attention Control. 2023b. http://///arxx wttp:/gg/bbb//......////
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference Computer Vision and Pattenn Recognition (CVPR), June 2016
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June: 1451- 11461, jan 2022. ISSN 10636919.
doi: 10. N0.1109/C9R 2022.0117222.. URL http://
arxiv. org/ abs 2201. 09865
Chenlin Meng. Yutong He, Yang Song. Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided Image Synthesis and Editing With Stochastic Differential Equations. ICLR 2022 10th International Conference on Learning Representations, 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text Inversion for Editing Real Images using Guided Diffusion Models. 2022. URL http://arxiv.org/
abs/2211.0972
Mohammad Sina Nabizadeh, Stephanie Wang, Ravi Ramamoorthi, and Albert Chern. Covector fluids. ACM Transactions on Graphics, 41(4):1-16, jul 2022. ISSN 15577368. doi: 10.1 3528223.3530120 URL https:///:d acm http://t.......rrrooo.././/1111. 11t.////1..1115322222333 353022
Haomiao Ni, Changhao Shi, Kai Li. Sharon X. Huang, and Martin Renqiang Min. Conditional Image-to-Videe Generation with Latent Flow Diffusion Models. mar 2023. URL http://
arxiv. Srxiv..rg//abs/20333 13744
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin. Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. Proceedings of Machine Learning Research, 162:16784 16804, 2022. ISSN 26403498. http:///://ax
org/ 2abb/21122 10741
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun Yan Zhu. Zero-shot Image-to-Image Translation. Proceedings SIGGRAPH 2023 Conference Papers, 2023. doi: 10. 10.114573588432355115533
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. 2022. URL http:://axxii/
abs/
2204.00 06125
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 0022-Jume:10674-110 dec 2022. ISSN 10636919, doi: 1LL1099CVVRR2228222222..0.11 abs fbs/2212..o 10752
URL http:://axxiv.ortt
Dan Ruta, Gemma Canet Tarrés, Andrew Gilbert, Eli Shechtman, Nicholas Kolkin, and John Col- lomosse, DIFF-NST: Diffusion Interleaving For deFormable Neural Style Transfer. 2023, URL
http::/aart /ttp:///axiv...// wrt.:ogg////.o.... absaabss///0/ .01157
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-Image Diffusion Models. pp. 1-10. Associa- tion for Computing Machinery (ACM), aug 2022. doi: 10. 10.1145//5222333333777..
Chachun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon. Edit-A-Video Single Video Editing with Object-Aware Consistency. 2023. URL httt:://axxiv.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Video: Text-to- Video Generation without Text- Video Data. sep 2022, doi: 10.48550/arxiv.220..477 147922.
URL htll:htt:://aat wwwp:/ivvorravooooooooooooooee
abs 14792
12

---

Published as a conference paper at ICLR 2024
Jascha SohhlDiccstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodyaamics. 32nd International Conference on Machine
Learning. ICML 2015, 5-226-22555 mar 2015.
URL http:/ aww.aaxiiv........ http:/ab/115000 1503.
03585
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling Through Stochastic Differential Equations. ICLR 202/ 9th International Conference on Learning Representations, nov 2021. URL http::///aaxi
org/abs/ 2011. 13456
Jingwei Tang, Vinieius C. Azevedo, Guillaume Cordonnier, and Barbara Solenthaler. Honey, I Shrunk the Domain: Frequency-aware Force Field Reduction for Efficient Fluids Optimiztionn Computer Graphics Forum, 40(2):339-353, may 2021. ISSN 14678659. doi: 10.f111/cgf.
142637. URL htl.https:/// http::///oonnneeebrbbee
wiley. com wttpccccc/cccccccccoc http///i/11111111/111111 1111/cgt egf 142637
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Eetio-EECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 1I 16, pp. 402-419. Springer, 2020.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-Play Diffusion Features for TexveDriven Image-to-Image Translation. 2022. URL http:://aa..w..oggabbb//////////
12572
Dani Valevski. Matan Kalman, Yossi Matias, and Yaniv Leviathan.
UniTune: Text-Driven Image Editing by Fine Tuning an Image Generation Model on Single Image. oct 2022. ISSN 15577368.
doi: 10.1145/3592455 URL httph://://axiiii http://awwxvvvvogaaaaaaa org.obgg///.o.o. abppg///bb/////aa///////////a 22210 09477)
Ruben Villegas. Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hemann Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable Length Video Generation From Open Domain Textual Description, pp. 1-17, 2022. URLhttp:
/ //arxiv. httpa//.rgg/bb/////aaeeea// 22100. 02399
Bram Wallace, Akash Gokul, and Nikhil Naik, EDICT: Exact Diffussinn Inversion via Coupled Transformations. nov 2022. URL
wttp://www...////////./111/ 22tp:/221111.1111111 12446
John B. Walsh. An introduction to stochastic partial differential equations. In École d' É'Été de Prob- abilités Saint Flour XIV- 1984. 2006. doi: 10.0007700000
Chen Henry Wu and Fernando De la Torre Unifying Diffusion Models' Latent Space, with Appli- cations to CycleDiffusion and Guidance. oct 2022. htt.http://aax...orr/bbb///// 05559
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-A-Video: One-Shot Tuning of Image Diffu- sion Models for Text-to-Vide Generation. 2022. URL http:://axxi..oot wrtp://///22//////////////////
11565
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the Generative Learning Trilemma With Denoising Diffusion Gans. ICLR 2022 - 10th International Conference on Learning Representa- fions, dec 2022. URL htRt:t/t:// arxi v. org/ abs 27.78044
Shuai Yang. Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender A Video: Zero-Shot Text- Guided Video-to- Video Translation. jun 2023. URL httt:://axxiv..o wttp:iiw..rr//bs//0000. 07954
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction a Large-scale Image Dataset using Deep Learning with Humans in the Loop. jun 2015. URL httt:://axiiv.oo/// www.rxiww..r.....oommmooo abs /lwp.////..... 1506. 03365
Guoqiang Zhang, J. P. Lewis, and W. Bastiaan Kleijn. Exact Diffusion Inversion via Bi-directional 200.222..
Integration Approximation. pp. 1-13, 2023. htt.ttt:://arx/
Lvmin Zhang and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. feb 2023. URL httt::::t///ae wwtp://wwwwwpppp/////2/0200 05543
13

---

Published as a conference paper at ICLR 2024
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. Inversion-Based Style Transfer with Diffusion Models, pp. 10146-10156, 2022. URL
http::/arxia org/ abs 2212 13203
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. ControlVideo: Adding Con- ditional Control for One Shot Text-to- rext-to-Video Editing. pp- 1-19, 2023. URL http::ttp://rttt
org hrg/aba/2en 2aoba/2305.110 17098
14

---

Published as a conference paper at ICLR 2024
A LIST OF SYMBOLS
D, (i,5)
Noise dimensions, pixel coordinates
G, G.
Discrete Gaussian noise sample (at frame n))
((i,)).
G(p)
Random variable obtained by evaluating G at pixel coordinates (i,j) or p
Xig, Z
Gaussian distributed variables
B(E)
Borel olaleebra on E, i.c. collection of all Borel sets on E
V
Standard Lebesgue measure, corresponds the area measure in IR²
EE)
o-finite measure space on E, with Borel o-algebra & = B)E) and measure y Continuous white Gaussian noise as set function
W
((A)
White noise defined over A (()), (AA) is Gaussian random variable Partitions of the domain E into regular grids different resolutions, A C E Number of per-pixel sub-pixel samples and noise variance for a k-th level a priori noise sample
4
A A
Nk. Uk
æ
p.
Σ
Discrete multivariable mean and variance
Mean operator
T
Deformation field used in warping
T(W)
Warping W with T using the noise transport equation, T(W) is white noise A subset of A i.e. 2p S A*
Qp
k, s
Noise up-sampling level, piecewise discretization level of pixel polygons
Table List of mathematical symbols used in the main text.
B CONDITIONAL WHITE NOISE SAMPLING
An important step in our pipeline is to lift a given initial discrete noise sample to a continuously defined white noise function. In Scction 2.1 we introduced a method for conditionally sampling white noise at any finer resolution given discrete sample. In this section outline additional important white noise properties; for a more rigorous white noise definition please refer to (Walsh) 2006 Dalang & Mountford 1999). We also include proof of Equation alongside with the pseudo- code for conditional sampling.
WHITE NOISE
Let F, be a complete probability space and let (E, v)) be o-finite measure space. White noise on E. based on is random set function
A W(A),
defined for E E with u(A) < co, with values in L2 F, P), such that:
1. ((A) is a Gaussian random variable with mean 0 and variance v(A);
2. if A and B are disjoint, then W (A) and W(B) independent and
W(AUBI B) = (AA) + W(B).
In our setting, we usually consider the continuous white noise to be defined on C R.. A useful fact about the covariance of ((A) and W W(B) is that can be computed from the definition:
((W(A)((BB) = (((W(A))) + W(An () ((B 4)))
= = =E((AAn B)
(8)
=r(AnB).
15

---

Published as a conference paper at ICLR 2024
B.2 CONDITIONAL PROBABILITY DERIVATION
Setup. To simplify the derivation and without loss of generality, let us consider a single discrete pixel associated with a Gaussian multivariable X (O,,, I1), where O, and I represent the zero and identity square matrices of dimensionality 1. In our noise formulation, the pixel does not represent a point, but rather defines a constant value in a unit square = [0, 1]2. Our goal is to subdivide the pixel area A into N x N sub-pixels, and sample them conditionally given an a priori sample X = x. To do so, we start by partitioning A into N2 sub-patches {Bx, such that
& x 1-1 (4....))(,.
1 W(Bk.) ~ NOO NZ). VA (L..E,,...,,NN N
B(BB.))
Proof of Equation 22. Let Y (W(BI,))..., (().))) be the random vector associated with all the sub-patches. Y is a multivariate Gaussian random vector thanks the independence of (WBB,,)) (BB,,)).. kt,1, and we have Y N(px = 022, by N= I=²). Now, consider the vector
Z (Y,X) From Equation (1) we have that X = ((A) = W (Us, Be..) - -En, W(Ba,), iee. X is a linear combination of Y. As a result, any linear combination elements in Z is also linear combination of elements in Y, thus Gaussian variable. Therefore, Z is multivariate Gaussian variable, and:
Z ~ (1A, E), with [Y 1X
C C (xxx)) C(x,x)
Σ =
(10)
where C constructs the covariance matrix for two given variables. Given the white noise definitions in Appendix B.I the covariance between two individual Gaussian random variables defined on A (pixel) and Вx, (sub-pixel) is their intersected area v (BR,1 O A) v(Br,) NT. This yields the following covariance matrices:
C(x,X) = NE IN2, C(Y,X) = Nou. u, C(X,Y) C(x,)), - C(xX)= = I1, (1I)
where u = (1...., 1 ER Since is Gaussian vector, the conditional probability of Y also a Gaussian, and its sampling is given by (Holt Nguyen 2023)
M(xxx) BY + C(Y C(x,) -1 "(x- (X-Pxl -xx): E(xix) = C(r.x) - C(r.x) (Cxx,))) C(x,))
Y N (F(XX)) E(x1x)), where
(12)
The terms above further simplify using Equation
as
1
M(YIx) = py + C(x,x) (Cxx,))) (X px) ONz
u(X
00, u,
N2
N2
(13)
1
1
E(([x) C(r,y) C(y,)) C(X
N2 IN= N
IN2
uu
N2
N2
N4
The last missing piece to derive Equation 2 comes by setting UUT E(xxx), with N (IN= Azuu") uu These definitions can be verified using the following identity:
(uu T)= uu uu = (u ujuu u
(14)
B.3 A PRIORI SAMPLING IN PYTHON
Unit variance scaling. Note that in the previous derivation, the upsampled noise represented by the new pixels (Bv,)) has a variance of 1/N2, Sometimes, it can be more practical to generate higher-
resolution discrete noise with unit variance. This is easily fixed by multiplying the Equation (3) by
N, resulting in the upsampling function
1
ups pm (x, = u (Z 12-1127 (Z)u), Z ~ (0,2 N
(Z) =
N2 u
(15)
16

---

Published as a conference paper at ICLR 2024
Algorithm I illustrates in Python procedure that upsamples H x W noise sample by factor N scaling it to unit variance.
Algorithm Conditional White Noise Sampling
def upsample noise (X, :):
b, e, h, w X. shape 2 = nerch.randie (b, e, N=h, N*w)
2_mean 2 (nffold N, N) unffcdd(3 N, NJ meen ((4, 5))
2_mean F. interpolate (Z_mean, ocale_factorw,, mode=' "ede=tnearest") X P.interpplate (x, saalefaactrrM, mode= 'nearest') return + mean
C DISTRIBUTION-PRESEVVIII NOISE WARPING
The distribution- p noise warping method proposed in Section 2.2 involves warping the continuous white noise with a diffeomorphic deformation field. In this section we mathematically derive the noise transport equation (Equation
C.1 BROWNIAN MOTION AND ITÔ INTEGRAL
White noise and Brownian motion. An alternative definition of white noise (x) bxee is through the distributional derivative of Brownian motion {B(x))xEE (also called Brownian sheet for dimension 2 2) as Wxx)dx = dB(()).
Itō integral. the Itô integral ʃA (x))d((x) of deterministic function @ T,2 is always Gaus- sian. The variance is given by:
E (fo())B(()) = d(x)) 110112.
(16)
From Equation
[6 we can relate back to the first definition of white noise by setting á Indeed,
W(A) = / 1dB(x)
(17)
XEA
a Gaussian variable of variance SxEA (xx)² SXEA 1dx v(A).
C.2 NOISE TRANSPORT EQUATION DERIVATION
Warping. The differential form of the transport equation for pointwise pixel values p(x) of an arbitrary image is given by
0p(x) -V -V)
(18)
Øt
where vxx velocity field. The mapping T is defined by integrating the of positions with the velocity field vxx). More specifically. T E + E is a diffeomorphism from our measure
space itself: T is differentiable, prrecture-rreeerving. and its inverse T-1 is defined for all x. The transport equation can be solved by the method of characteristics mapping (Nabizadeh et al.) 2022)
generate the warped field p(x) by
p(x) = (x)).
(19)
However, this method only considers pointwise pixel values, whereas our definition of white noise requires the transport in continuously defined pixel areas. In 2-D this amounts regularizing point- wise values by the area that they represent, while simultancously tracking the distortions resulting from the mapping application. This notion is also common in exterior calculus: ()-frrms (points) to
17

---

Published as a conference paper at ICLR 2024
2-forms (faces) are connected through the dual operator (Crane 2013). volume method to represent p(Ax) as a cell-centered area averagee integration
We employ simpler finite
(( = v(Ax)
p(x) dx.
(20)
XEAx
Note that the resulting p(Ax) can still be considered pointwise quantity, since it is the integrated
value over pre-specified area regularized by the area size. Applying Equation 20 to T (Ax), the warped cell-centered area quantity x(Ax) is
x(Ax) ((T VIT J
p(x) dx.
(21)
XXX-
ones naively replaces by a white noise distribution W, the warped noise W follows
(Ax)= u(T v(T-I(Ax)) xxX- W(x) dx.
(22)
((Ax) is Gaussian random variable, since is Itô integral. Its variance is given by evaluating Equation (16) as
I XET-
(((x)) u(T-I(Ax))²
12 dx u(T-I(Ax)
(23)
The variance of the integrated white noise above is correct, it should be proportional to the integrated area, in accordance with the definitions in Appendix B.T This shows that a continuous Gaussian white noise loses its distribution when we transport with standard transport equations. In Section 2.2, we introduced an alternative to the standard transport equations which accounts for variance preservation. We referred to it as the noise transport equation
((A) = f
XEEA VT (T- 3 W(T (x)) dx.
(24)
defined for any A C E. This equation correctly accounts for a variance rescaling term, which ensures that the integrated white noise is in conformity with its defining properties. We prove that Equation (24) is the correct formulation that produces continuously defined noise representation that further satisfies
1. W is a white noise as defined in Section B.1
2. W is a warping of W by T, i.e. it satisfies relation similar to Equation 19 Specifically,
W(Ax) (AX)) v ((Ax)= (Ax-)
(25)
Proof 1. To show that W is indeed a white noise, we first show that for A s.t. v(A) < 00, we have (AA) _N(O, v(A)). We proceed change of variable y Tx),x), i.e. (T(y)) dx. Thus,
((A)
(AT [T(y) (
[VT (y)dd
(26)
(ET-)(())
(d(y))
YET 1(A)
Again, this Itō integral is Gaussian variable and its variance is given by:
18

---

Published as a conference paper at ICLR 2024
dy (IVT(y))
yET-1(A)
VT
(27)
-YET-1(A)
dx= v(A),
AXEA
where we used the change of variable the other way round. It is not difficult to verify that W also satisfies the second condition on disjoint sets, thanks the linearity of the integral operator and the bijectivity of T.
L
Proof Next, we want to prove that this new noise corresponds to our intuitive definition of warped noise. For this, let x E E and consider Ax E & a tiny subset around x. By setting A T(A)) in the definition of W, we have:
/ (YETIAN) (T((T+(y)) W(T W(T-(vy)) dy))
W(T(Ax))
(28)
By the same change of variable as before we get:
W(T(Ax)) = / yEAx VT (y)))
(29)
By linearization, we assume that the determinant of the warping field's Jacobian is constant over Ax, replacing it by its mean yields
VT (u) |du 1
W(T(Ax)) /
/
W(y)dy
JYEAx ((Ax))
JEEx
(V W(r)dd)
VT (u) |du
v((x) Jueax
(30)
$ (JrEae dB(r)) dv (UT(AA))) W(Ax), v(Ax)
((Ax) VET(AX)
which concludes the proof.
C.3 FROM CONTINUOU'S TO DISCRETE FORMULATION
In practice, it is infeasible to warp a continuous noise field with the noise transport equation. We will demonstrate that Equation (4) becomes Equation 5 once discretized. Our goal is to compute the pixel value G(p) the warped noise for the given pixel coordinates [-nose interpretation, G((p) corresponds
= (i, Using our the integral the underlying warped white noise, i.c.
G(p) = (A), where A = [i 1, x [j - 1, is the subset of the domain E covered by the pixel at position p. Thus, substituting G(p) in Equation (4):
/
G(p) = |XEA VT (T W(T-1(x))
(31)
Assuming that the warping field has a locally constant Jacobian over the entire pixel area around p. apply the approximation from Equation (25) with Ax A:
19

---

Published as a conference paper at ICLR 2024
v(A) 3 U(T ((A) v(77(11)) W(T-
G(p)
(32)
(AT-2 W(T-
since v(A) 1 by definition of A.
The second approximation comes from rasterization. We approximate the warped pixel shape by its rasterized version at level k. This is formally written as
(A)
U 1 AN
(33)
where Spp A contains all subpixels at level &o that covered by the warped pixel polygon (A) after rasterization. Substituting this into (()) and using linearity of and W, we get
G(p) = v (UD
W
U Bp
14)
Σ
w(A
EatEr ((a)))
AgED,
(34)
NR Σ W(AA)
since v(A,) 1/NB>
V 12pl
AEER
Σ AtE
wx(AF)
since Wx(A{) VNEW(A)),
12p
which concludes the derivation.
is easy to verify that the resulting G(p) remains a standard normal Gaussian variable, as it the "Gaussian average" of multiple standard normal variables. Furthermore, the independence between different pixels ((p) and G(q)) is ensured with a proper implementation of rasterization, i.e. rasterization of adjacent triangles are disjoint.
Lastly, note that all the approximations we made to go from the continuous setting to the discrete one only affect the accuracy of the deformation field (locally constant Jacobian, approximate warped region), but never the actual values of the noise samples. This is key to the distribution-preserviin properties of our method.
C.4 IMPLEMENTATION
We include the pseudo-code of our f-noise warping for single pixel in Algorithm (2). and provide more details on the algorithm itself regarding specific aspects such as warping and the treatment of disocclusions.
Warping. To warp the triangulation points V for the pixel contour, we follow the common practice in physics simulations. As the deformation map T only defined at pixel centers, we use bicubic interpolation to obtain the deformation map values at the sub-pixel triangulation points.
Handling disocclusions. When employing optical flow maps as the deformation mapping T be- tween the anchor and current frame, some pixels can end up with an undefined value. There two possible causes to this issue. The first one is that the pixel polygon was stretched in a way that it ended up covering zero sub-pixels after rasterization. This can happen when the deformation is large, or the sub-pixels are not fine enough (i.e. is too low). The second one that two pixel poly- gons rasterized one top of the other. As only keep the values of the last one, sub-pixet
20

---

Published as a conference paper at ICLR 2024
Algorithm 2 Distribution-preseree noise warping (for single pixel)
Input: G: discrete noise anchor frame (in size D D)
A: pixel area in current frame
deformation mapping between the anchor and current frame
k: noise upsampling factor
5: polygon subdivision steps
Output: pixel value x: in current frame
Go UPSAMPLE (G,
FV +- TRIANGULATE
V < (V
Ω - RASTERIZE((V, F), Goc)
- Duwlen Golli.)) (SIZE(S))
associated with the first polygon. This typically happens if there are disocclusions and the flow map is not diffeomorphic.
To solve this in practice, we use a two-stage process. First, we warp the noise from the initial frame and fill in the current frame's pixels. At this stage, some pixels might be undefined. At the second stage, we perform another warping, this time from the previous frame (up-sampled on the fly), and use fill the remaining missing values. If there still missing values left, replace them by randomly sampled noise. This allows the best temporal consistency for most of the pixels.
C.5
1-D TOY EXAMPLE: PROOF
In Section 2.2 we gave an intuition on why our ['noiee is able preserve the distribution using 1-D toy example. Here we aim at proving Equation 6 in similar way of proving the conditional white noise sampling presented in Appendix B.2
First, let us consider continuously defined noise signal W over the 1-D segment E = [0. 2]. We further define two independent random variables X2 [[0,, 1]), x1 W 2)) representing two neighboring "pixels". A random variable interpolating linearly between xo and in space can be defined as z W ([a,, + al), with a E [0, 1]. Then, all three variables have a variance of and is correlated with X and x1 through
Cov(z, xn) = v((a, + a] n [0, 1]) v[aa, 1)) = 1 a.
(35)
Cov(z,x1) v((a, 1 + [1, 2]) v(11, al) a.
Thus, merging the variables into vector Y = (z, Xo, x yields
1 -
Y (M, E), with μ 0, Σ
a
(36)
1
Applying the conditional probability rule for Gaussian vector (as in Appendix
B.2
gives:
fū= = (1 a, a) (x0. x1)T
z|xo = Zo, I N (2, where
(37)
T2 = (1 .11 a,,
Simplifying the expressions yields
e = (1 a) az1, and o² - (a² (1 (1)
(38)
which concludes the proof.
C.6 COMPARISONS AGAINST OTHER WARPING APPROACHES
Interpolation baselines. Our method enables accurate Gaussian noise warping, even when the noise undergoes substantial deformations. Given two dimensional warping field extracted from
21

---

Published as a conference paper at ICLR 2024
Frame 5
Frame 35
Frame 65
Bilinear
Cubic
Nearest
Root-bilincar
f-noise (ours)
Figure 7: Warping a noise sample with different schemes. Rows indicate different frames, while columns represent the method used for transport. The noise sequence is warped incrementally, transporting one frame at the time with a given velocity field. Warping through bilinear or bicubic interpolation quickly blurs the noise image. Nearest neighbor produces better results, but introduces artifacts due to incorrect spatial correlations. Root-bilinear interpolation has the proper distribution start, but produces incorrect results once the pixels are no longer independent. Our proposed J noise is able to properly maintain temporal correlations introduced by warping while also preserving the original signal distribution.
the fluid simulation dataset (Figure Figure shows a comparative study between various warping methods. Three classical types of interpolation were tested: bilinear, bicubic and nearest neighbor. While bilinear and bicubic interpolations compromise high- high-frequency details, nearest neighbor cre- ates artifacts in the form of duplicate samples, since it is oblivious to space deformations introduced by the warping field. We also experimented with a simple modification the bilinear interpola- tion that is able to preserve the variance of the pixels by replacing the weighting coefficients with their square roots. Though being efficient, this approach still introduces spatial correlations that also eventually lead to artifacts. Our noise method outperforms all existing warping methods by transporting the noise perfectly while keeping its Gaussian properties.
Noise prior baselines. We also compared the proposed [ -noise warping against previous noise priors such as the residual-based noise sampling of Control-A- Video (Chen et al. 2023). and the mixed and progressive noise model from PYoCo (Ge et al.] 2023). The residual-based sampling keeps the noise signal fixed for image pixels that remain unchanged in video, while the noise resampled in locations in which temporal variations of RGB values go over predefined threshold. PYoCo introduces two sampling schemes: mixed and progressive. Mixed noise sampling linearly combines frame-dependent noise sample and a noise shared across all frames. Progressive noise sampling generates sequence of noise samples in an autoregressive fashion: the noise at the current frame is generated by perturbing the noise from the previous frame. These approaches are not able to preserve the exact spatial correlations of the original sequence, since they are not based on warping. We show comparison between different warping methods for the task of super-resolution
22

---

Published as a conference paper at ICLR 2024
8 2
8 3
8 4
Method
Wall Time
CPU Time
21.6
21.7
26.2
Random
0.01
0.01
k
23.6
23.5
23.8
25.1
0.01
k 2
30.5
29.3
29.6
30.8
PYoCo (mixed)
0,01
0.01
k 58.8
55.4
55.0
53.7
PYoCo (prog.)
0.01
0.01
1: 143.8
137.5
132.4
128.2
Control CoVieo
6.08
95.46
Bilinear
5.26
76.76
Bicubic
6.00
87.73
k 10.5
10.6
11.3
12.9
Nearest
5.17
75.73
k 10.5
10.6
10.8
11.3
Root-bilincar
7.66
103.78
k 11.2
11.3
11.5
11.9
DDIM Inv. (20 Steps)
853.42
2226.6
4: 15.5
DDIM Inv. (50 Steps)
2125.5
3608.3
4: k=
29.1
29.2
28.6
28.7
-nose (ours, 3,
629.01
2274.9
Table 4: CPU time (top) and wall time (bot- tom) of f-noise computation for different and s parameters. The measurements vidco sequence of 24 frames resolution 256 x 256, in seconds.
Table Runtime Comparisons of Different Noise Schemes. The measurements are in milliseconds per frame resolutions 256 x 256.
in Figures 14 and T5 We refer our supplementary website for additional comparisons between different warping techniques.
C.7 RUNTIME COMPARISONS
We estimate both wall time and CPU time for all methods on video sequence at resolution 256x in Table 3 By evaluating the run time for different sub-sequence lengths, we can linearly regress the average per frame computation time of each method. This is evaluated GeForce RTX 3090 GPU and an Intel 19-12900K CPU.
Simple noise sampling methods like PYoCo Ge al. (2023), fixed or random noise can be executed efficiently on GPU. Methods that rely on information from the input sequence in a simple way such Control-A- Video Chen et al. (2023) the interpolation methods are 3 orders of magnitude slower than the simple noise sampling methods. Finally, our warping method requires second per frame to warp the noise. While being considerably less efficient than aforementioned methods, our approach is comparable to DDIM inversion.
Additionally, method trades off the noise warp accuracy with efficiency through two parameters: k. the noise upsampling factor, and 8, the polygon subdivision steps. We report the CPU and wall clock time in seconds for noise warping in video sequence of 24 frames resolution 256 256 for different & and in Table
D ADDITIONAL RESULTS
In this section, we provide more results and evaluations for the applications shown in Section 3
D.1 PHOTOREALISTIE APPEARANCE TRANSFER WITH SDEDIT
Figure 8 shows different variations of photorealistic appearance transfer from the same synthetic video sequence (top row). Contrary DDIM inversion, our temporally-correlan warping scheme can be applied to arbitrary noise samples, allowing to generate different edits from the same input sequence. In Figure [0 we compare our -noise warping against fixed noise sample on a close-up scene. When using lixed noise sample, the original fold in the corner of the bed stays at the same position in the image even after the bed rotates; our method, on the other hand, successfully trans- shown
ports the fold, following the bed movement. Other comparisons for the bedroom example in Figure and [2 Table 5) we quantitatively evaluate our method on different bedroom scenes. Our noise warping method not only outperforms standard Gaussian noise priors in terms of temporal
and
23

---

Published as a conference paper at ICLR 2024
Method
w/o cross-frame attention
w/ cross-frame attention
seene
scene
scene
mean
scene
seene 2
scene 3
mean
Random
12.85
11.82
11.21
11.96
10.02
10.23
9.74
10.00
Fixed
4.75
4.19
5,44
4.79
4.40
3.90
4.74
4.35
PYoCo (mixed)
9.25
8.28
8.44
8.65
7.67
6.79
7.31
7.26
PYoCo (prog.)
6.27
5.52
6.41
6.06
5.18
4.26
5.25
4.90
Control-A --e--A-Viie Video
5.13
5,48
6,08
5.56
4.65
4.71
5,91
5,09
Bilincar
3.12
3,93
8.33
5.13
2.70
1.89
4,86
3.15
Bicubic
3.24
5.97
15.10
8.10
2.67
3.05
9.12
4.95
Nearest
13.77
28.78
46.59
29.71
5.12
13.69
26.48
15.10
[-noise (ours)
3.47
2.09
3.68
3.08
2.97
1.70
2.83
2.50
Table 5: Temporal coherency metries for the task of appearance transfer with SDEdit. Each value is averaged over batch of 4 different variations. The metric is the masked MSE between the current frame and the warped previous frame averaged over the entire sequence (units are in x 10-3).
Method
w/o cross-frame attention
w/ cross-frame attention
FID
Precision
FID
Precision
Random
61.14
0.659
74.75
0.719
Fixed
79.32
0.622
93.56
0.644
PYoCo (mixed)
65.07
0.656
81.60
0.674
PYoCo (prog.)
66.43
0.635
84.63
0.667
Control-A- Video
75.21
0.618
90.82
0.649
Bilinear
185.28
0.101
143.24
0.201
Bicubic
168.86
0.118
149.85
0.212
Nearest
192.13
0.177
154.73
0.344
-noise (ours)
73.83
0.655
92.63
0.661
Table 6: Sample quality metries for the task of appearance transfer with SDEdit. We measure perceptual quality of the generated bedroom frames. The FID (Heusel al. 2017) metric measures sample quality and diversity at the same time. The precision metric (Kynkäänniemi et al. 2019) measures sample quality through the likelihood of our generated samples belong the LSUN Bedroom dataset. We use the implementation from Dhariwal & Nichol (2021).
coherency, it also manages to beat standard interpolation methods. The reason for this is visualized in Figure 11 standard interpolation methods are unable to account for space deformations, which leads to artifacts in the denoised results.
We evaluate the visual quality of the generated frames in Table Gussing FID (Heusel et al.) 2017 and Improved Precision (Kynkäänniemi et al.] 2019). FID (Heusel et al.) 2017) is computed between the LSUN Bedroom dataset and the set containing all images from a given video example. It is important to note that FID simultaneously measures sample quality and diversity, while we are only interested in quality. Since we are computing FID over sequence of frames from the same video, the diversity is naturally very low, which explains the overall high values we have in T FID values are less than 50), Additionally, less temporally coherent methods like Random Noise will likely have more variations between frames, which improves diversity. This explains why random noise performs the best in terms of FID, and our method typically performs slightly worse All in all, the main take-away from the FID comparison is that our method performs in the same range other Gaussian noise priors (random, fixed, PYoCo, Control-A- A-vrdo)))VVdeo) and much better than the other interpolation methods. Improved Precision (Kynkäänniemi et al. 2019) a metric of sample quality only, it measures the likelihood of the generated samples belong to the ground truth dataset manifold. As the table shows, our method performs on par with other Gaussian noise priors in terms of visual quality.
We additionally compared our noise prior with DDIM inversion in Figure B As DDIM cannot be directly used with DDPM-based methods like SDEdit, devise reasonable ways of comparing
24

---

Published as a conference paper at ICLR 2024
Figure 8: In combination with cross-frame attention, noise warping helps create temporally-coheree realistic rendering room from a synthetic scene. Top row: input synthetic scene, other rows: results obtained using f -noise with different initial noise.
Figure 9: While random noise creates different details each frame (top), fixing the noise leads to the exact same details in image space (middle). Noise warping successfully translates the details with the object (bottom).
25

---

Published as a conference paper at ICLR 2024
Figure 10: Using the same noise at every frame (top) leads to sticking artifacts, as shown by the bedsheet fold staying in the same spot of the image. Noise warping mitigates that (bottom).
Bicubic
Bilinear
Nearest
Ours
Figure 11: Our method has quantitatively better temporal cohereney than traditional warping meth- ods, since large motions cause unaccounted noise stretching for classical interpolation (top row), which leads to visible artifacts the denoised image (bottom row).
Figure 12: Comparing f -noise (top), fixed noise (center), and random noise (bottom) for different bedroom example.
26

---

Published as a conference paper at ICLR 2024
Sample
Sample
Sample 3
Sample
Noisc
Figure 13: Comparing DDIM inversion to intermediate step (top), DDIM inversion initial noise (middle), and our ofr--noof |nnois prior (bottom).
with DDIM noise in photorealistic appearance transfer. the first experiment (Figure 13 top row), apply DDIM inversion intermediate step (60% total steps) and then denoise it using forward DDIM. As expected, since neither the prompt or settings are changed, this approach mostly recon- structs the original synthetic video without adding any realistic appearance details. In the second experiment, DDIM inversion is employed to obtain the initial noise (Figure [3] middle row), we then add to input frames and denoise using forward DDIM. Because distribution of the model (trained on realistic images of bedrooms), the DDIM-inverted noise is far from Gaussian. This makes it poor candidate as a noise prior. As DDIM remains primarily an version method, the spatial and temporal information of the image are entangled inside the noise. On the contrary, our noise prior only contains temporal information, so the model can generate realistic details top of the synthetic scene without being constrained to reconstruct the input sequence.
input video far from the data
D.2 VIDEO RESTORATION WITH 12SB
We evaluate our noise prior on two video restoration tasks based on 12BB namely super-resolution and JPEG compression restoration. Table respectively evaluate the temporal coherency and the image quality for different noise priors. Our -noise prior performs worse in JPEG restoration
Warp MSE (x10-3)
4x super-resolution blackswan
JPEG restoration
bear
car turn
mean
COWS
goat
train
mean
Random
3.98
9.57
13.17
8.91
6.71
11.36
6.78
8.28
Fixed
2.18
8.69
13.03
7.97
5.15
11.21
6.25
7.54
PYoCo (mixed)
3.09
9.28
13.08
8.48
5.84
11.12
6.35
7.77
PYoCo (prog.)
2.32
9.08
12.90
8.10
5.33
11.05
6.06
7.48
Control-A- A-Vico
2.24
8.88
12.82
7.98
5.73
11.16
6.31
7.73
Bilinear
9.96
25.52
28.94
21.47
2.85
8.44
4.49
5.26
Bicubic
4.60
15.79
18.66
13.02
3.34
Nearest
2.95
17.38
22.56
14.30
5.03
11.14
7.55
7.91
J -noise (ours) 1.19
6.76
11.51
6.49
4.04
8.88
5.01
5.98
Table 7: Temporal coherency analysis for ISSB video tasks. We show temporal coherency results for SB video tasks evaluated on the DAVIS dataset. The metric the masked MSE between the current frame and the warped previous frame averaged over the entire sequence (units in x10-3).
27

---

Published as a conference paper at ICLR 2024
PfoCo
PfoCo prog.
Rnd
Fix
C.v.V.
Bilin
Cubic
NN
Ours
ProCo mix.
PfoCo
Rnd
Fix
C.a.V. V.
Bilin
Cubic
NN
Ours
PioCo PfoCo prog.
mix
ca.vv
Cubic
Ours
Car turn
Bear & Blackswan
Figure 14: Visualizing z-t slices of the I'SB video super-resolution task. We visualize the differ- ent methods from the quantitative evaluation (in the same order). Our noise prior (right) does not exhibits stripes that are manifestations of sticking artifacts. also seems less blurry than standard interpolation methods, and less noisy than Random Noise.
relative to super-resolution. This might be due to the optical flow being harder to estimate from the corrupted sequence the first case.
In Figure 14 and Figure 15 we visualize temporal slices of the denoised videos by concatenating single column from each frame. this visualization, sticking artifacts manifests as horizontal stripes, making it easy to evaluate temporal coherency qualities. The various sequences show that warping the noise consistently yields temporally-smoother and spatially-sharper results.
LPIPS
4x super-resolution
JPEG restoration
blackswan
car turn
mean
COWS
goat
train
mean
Random
0.242
0.179
0.154
0.192
0.157
0.201
0.131
0.163
Fixed
0.224
0.180
0.134
0.179
0.151
0.201
0.136
0.163
PYoCo (mixed)
0.242
0.179
0.149
0.190
0.156
0.207
0.136
0.166
PYoCo (prog.)
0.243
0.172
0.156
0.190
0.153
0.205
0.132
0.163
Control-A- Video
0,240
0.183
0.154
0.192
0.156
0.205
0.132
0.164
Bilinear
0.668
0.589
0.315
0.590
0.447
00.53
0.332
0.431
Bicubic
0.627
0.500
0.343
0.490
0.358
0.452
0.305
0.372
Nearest
0.297
0.414
0.275
0.329
0.167
0.253
0.221
0.213
J-noise (ours)
0.235
0.202
0.152
0.196
0.154
0.202
0.138
0.165
Table Image perceptual quality analysis for FSB video tasks. We measure perceptual quality of the generated results for I'SB video tasks evaluated on sequences from the DAVIS dataset. The use LPIPS (L) with VGG network assess the visual quality of the images. Standard interpolation methods leads to blurred results, which affect the perceptual quality. Our noise prior performs in the same range as standard Gaussian noise.
D.3 POSE-TO-PERSOX VIDEO GENERATION WITH PIDM
When employing the Person Image Diffusion Model (PIDM) Bhhnnia et al. 2022) for generating videos relative to input posed person sequence, the pose sequence not enough to estimate accu- rate motion vectors. So we run a few steps of diffusion with PIDM to get a blurry early prediction of the person, from which we then extract the full body pose with DensePose (Güler al. 2018). The
28

---

Published as a conference paper at ICLR 2024
Train
Rnd
mik.
prog
C.a.V.
Cubic
NN
Ours
ProCo mix.
ProCo
ProCo mix.
PfoCo
Rnd
Fix
C.v.V.
Bilin
Cubic
NN
Ours
Rnd
Fix
C. V.
Bilin
Cubic
NN
Ours
Cows
Goat
Figure 15: Visualizing z-1 slices of the ISS video JPEG restoration task.
Pose
Early prediction
DensePose
Final prediction
Figure 16: Pipeline for motion estimation using PIDM.
DensePose result is overlayed on top of the early prediction the final prediction before computing the optical flow (Teed & Deng 2020). The different stages of the pipeline are depicted in Figure I6
We get test examples from the DeepFashion dataset (Liu et 2016). Table gives more detailed quantitative evaluation regarding temporal coherency. In particular, we notice that in the examples evaluated in Section 3 the pose sequences generally have little motion (as is often the case when fashion models pose). This could explain why fixing the seed has a better temporal coherency. We alternatively test on pose sequences where we simulate the camera moving, causing the poses to shift horizontally, The results are reported in the table under "High-varians pose". Similarly, fixing the random seed gives fairly good temporal coherency. This is in contradiction with the visual results, which clearly shows sticking artifacts. A possible reason comes from the fact that compute the temporal coherency on the entire image, and most of the background has zero motion, which may bias the mean warp error favor the fixed seed experiments.
29

---

Published as a conference paper at ICLR 2024
Warp MSE (x107))
Quasi-static pose
High-variation pose
input
input 2
input 3
mean
input
input 2
input 3
mean
Random
26.40
46.50
23.72
32.21
27.01
50.13
25.53
34.22
Fixed
2.40
3.29 e
3.14 e
2.94
1.76
2.22
2.78
2.26
PYoCo (mixed)
15.81
22.46
12.43
16.90
18.78
28.73
15.25
20.92
PYoCo (prog.)
9.73
15.11
8.28
11.04
11.92
15.06
8.91
11.97
Control-A-Vicn
6.34
13.90
8.47
9.57
6.41
7.00
5.94
6.45
Bilinear
1.77
3.00
2.87
2.55
1.83
2.30
2.28
2.13
Bicubic
3.15
4.07
4.10
3.77
2.15
2.41
2.66
2,40
Nearest
11.69
17.83
12.73
14.08
7.21
13.21
7.20
9.21
[noise (ours)
2.54
4.58
3.52
3.55
2.42
3.03
3.29
2.92
Table 9: Temporal coherency analysis for the pose-to-person video with PIDM task. We show temporal coherency results for pose-to-person video with PIDM using input images from the Deep- Fashion dataset.
D.4 FLUID SUPER-RESOLUTION
Our last application is on the super-resolution of 2-D fluid simulations. We train unconditional diffusion model on a large dataset of fluid simulations generated with the fluid solver from Tang et al. (021 without special treatment the noise. We then compare different noise priors, including the one from Chen et al. (023). The results are hard to assess frame by frame. Thus, we encourage the reader take look the supplementary videos.
E NOISE WARPING IN LATENT DIFFUSION MODELS
E.1 UNDERSTANDING THE LIMITATIONS
Noise warping does not improve temporal coherency in latent diffusion models as much as one would expect. We pinpoint this observation to three main reasons. First, temporally coherent images do not necessarily translate into temporally coherent autoencoder latent vectors. Thus, temporally consistent noise priors in the VAE latent space might be suboptimal. Second, the noise used in latent diffusion models does not contribute directly to the fine details of the results in image space, which is where warped noise priors excel. Lastly, the autoencoder almost always introduces a non- negligible reconstruction error that affects the temporal coherency of the results, independent of the latent diffusion process.
From a more thcoretical perspective, the comporally-coreekae infinite-resolution noise warping propose is only possible because of the structural self-similarity of Gaussian noise, which allows us interpret discrete Gaussian noise sample as an aggregated view of more Gaussian samples at smaller scale. The warping could thus be operated at the limit, in the continuous setting. This core assumption does not hold in latent diffusion models, because temporal coherency is no longer targeted in the latent space where the noise resides, but rather in the image space mapped by the decoder. Unfortunately, Gaussian noise decoded by the VAE no longer possesses the self-similarity property. This aspect be visualized in the decoded noise shown by Figure 17al (a) (ii)..
Nonetheless, the assumption that moving the noise helps with temporal coherency still holds in latent models to some extent, some of our experiments below can show. One of our early observations was that the VAE decoder is translationally equivariant in a discrete way, i.e. translating the latent vector by integer number of pixels leads to almost perfectly shifted image when decoded. For an autoencoder with compression factor f = (c.g. Stable Diffusion), this means that a I 1-pixel shift in the latent space would produce 8 8-pixel shift in the final image.
30

---

Published as a conference paper at ICLR 2024
(ii)
(iii)
(a) Inputs to the latent diffusion experiment: (i) input frames, (ii) warped latent noise, (iii) decoded by VAE.
rand.
fix.
warp.
(b) Vidco-to- toiidcoo translation baseline: frame frame w/ ControlNet and prompt.
warp.
(c) Video-to-vide translation: (b) w/ fross-frame attention.
warp. lix.
(d) Video to-video translation: (c) feature injection.
Figure 17: Noise warping in latent diffusion models. As the latent diffusion model gets more constrained, through ControlNet and prompt (b), cross-frame attention (c), + feature map injec- tion (d), results improve temporal coherency. In the final example (d), translating the noise along with the content gives a pixel-perfect translating pizza, while keeping the noise fixed still induces sticking artifacts on the high-frequency details. Please refer to the supplemental videos for better visualization of the differences between these examples.
31

---

Published as a conference paper at ICLR 2024
E.2 EXPERIMENT SETUP
With this in mind, we generated a simple video consisting of two concentric gray discs translating by 8 pixels the right every frame, in front of a white background (see Figure (a) (i). We add noise to each frame, and denoise it using Stable Diffusion with the following prompt:
"Commercial photography, square salami pizza, italian food, white lighting, top view, studio light, 8k octane rendering, high resolution photography, insanely detailed, fine details, on white isolated plain, 8k, commercial photography, stock photo, professional color grading"
We manually simulate the warped Gaussian noise used in latent space by moving a disc of noise in front of noisy background (Figure 17a (a) (ii).. The input frames are used "scribbles" with Con- trolNet and the prompt provides the necessary information to generate the results. This sequence, as many of our other results, better visualized the included supplementary videos.
E.3 RESULTS AND ANALYSIS
We perform three sets of experiments, each time with increasing constraints the model. All the results reveal empirically that our f prior is better, provided it is warped properly.
using the input frames with ControlNet and the given prompt, we generate results frame by frame (Figure 17b). While all the noise schemes produce temporally inconsistent results, flick- ering is especially visible when using random noise at each frame (top row). For fixed random seed (middle row), the insets show an overall structure is fixed in image space, even though the pizza is translating. This can be seen be paying attention to the regions covered by the basil leaves, which remains rougly the same. Finally, warping the noise with the input video improves the movement of the synthesized details (bottom row). In particular, the basil leaves progressively invades the inset, indicating proper translation.
Baseline:
Cross-frame attention: a common trick used to boost temporal-coherency in diffusion- video- to-video translation tasks is to replace all self-attention layers by cross-attention with respect to an anchor keyframe (Hertz et al., 2022; Parmar et al., 2023). Figure[[7]] shows the results of addition- ally adding cross-frame attention with respect to the first frame for every frame. This effectively creates more constraints for the model, leading to similar content between frames. Nonetheless, fixing the noise (top row) still creates sticking artifacts, mainly the position of the salami slices in image space. Another visible artifact can be seen in the position the darker spots the pizza crust, which stays in place. On the contrary, moving the noise with the input generates much more natural motion the ingredients on the pizza (bottom row).
Feature map injection: another tool for improving temporal coherency involves injecting feature maps (Tumanyan et al.] [2022) from one image to the other. This is usually done in addition to employing cross-frame attention. It can be applied between frames, as long as feature maps also warped. By injecting the feature maps from the first frame for all layers 4-11, our noise warping method produces pixel-perfect translation of the generated pizza (Figure [id bottom row). Keeping the noise fixed, however, induces sticking artifacts of fine details (top row).
Analysis. As shown by the experiments, the impact of the noise scheme is negatively correlated with the amount of constraints given to the model. As most of the low-frequency structures gets dictated by the cross-frame attention and the feature maps, the noise becomes solely responsible for the fine high-frequency details. In any case, noise warping is a necessary treatment tor provide temporal coherency to diffusion models when they are employed on frame by frame basis.
32


Reviews:
[{"review_id": "JgyloWVjJY", "reviewer": "Reviewer_13WQ", "summary": "This paper introduces the novel problem of Gaussian noise warping. Therein, the authors raised the interesting question of \"how to properly warp the Gaussian noise such that the warped noise map still has the Gaussian distribution preserved?\". To this end, the authors discussed why applying conventional warping operation on the noise map is not suitable, and they proposed a mathematically grounded solution to the problem with their ∫-noise formulation. The authors motivated the practical value of this problem with the applications of lifting image diffusion models to perform temporally consistent video editing. Experiment results with different video editing tasks demonstrate the effectiveness of the proposed method.", "questions": "Other than the comments above, there are a couple of questions I’m curious about:\n+ I’m wondering how sensitive the method is with respect to the underlying estimated flow map? In other words, how do the errors in the estimated optical flows affect the results?\n+ One principled way to obtain a sequence of correlated noise maps is to perform inverse DDIM on the reference video frames (assuming such video is available, which is true in most of the use cases demonstrated in this paper). The authors did mention that technique in the paper, but did not elaborate further and did not show visual results or comparison. I’m wondering how will the noise maps obtained that way compare to the ones obtained from the ∫-noise in terms of the final video quality?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The research question “how to properly warp a Gaussian noise map” introduced by this paper is interesting scientifically.\n\nThe problem is well motivated with clear explanations on why the problem is not trivial, i.e. why the conventional warping operations are not suitable. The proposed solution is technically sound and well-grounded mathematically.\n\nThe practical aspects of the problem are also well-motivated, with interesting video editing applications.", "weaknesses": "The major weakness from the practical point of view is the implicit assumption that temporally correlated noise maps can induce temporally consistent video editing results, which is often not true. This limitation, however, has been acknowledged and explained by the authors in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2aJQPL7qKT", "reviewer": "Reviewer_3JNo", "summary": "This work attempts to mitigate the lack of temporal correlation seen in diffusion based video generation models. The proposed method attempts to generate new noise samples that preserve the correlations induced by motion vectors. To this end, first, this work reinterprets individual noise samples used in diffusion models as a continuously integrated noise field called integral noise. With the help of the derived noise transport equation, a transport algorithm is developed to generate noise with temporal correlation between samples while preserving the desired properties of noise samples. Results are demonstrated to indicate the potential of the proposed method for tasks such as video restoration and editing, surrogate rendering, and conditional video generation.", "questions": "1. Figure 5 illustration is hard to follow, it’s not clear how to judge the performance based on the images shown. It’s mentioned that, the Random Noise creates incoherent details while Fixed Noise suffers from sticking artefacts. Our R -noise moves the fluid in a smoother way. However, I find this explanation hard to map to the results shown. Maybe the authors can highlight based on the image contents the reasons for each of these conclusions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem tackled in this work has significant practical impact.\n2. The paper is written very well.\n3. Motivation is clear, ideas are well formulated with theory, and the results are impressive.", "weaknesses": "1. The proposed method is computationally inefficient as compared to prior arts. Quantitative comparisons on this aspect would have been more helpful for future research works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOCBTdMMgw", "reviewer": "Reviewer_H5Nc", "summary": "This paper proposes a method for warping Gaussian noise while preserving its Gaussian properties. \nTo achieve the goal, the authors consider a mathematical model of the Brownian sheet (i.e. the distributional derivative of the two-dimensional Brownian motion) and develop a noise transport equation for that model. \nIn applications, the following approximation is applied. First, the input noise map is conditionally up-scaled so that it remains Gaussian. Second, the up-scaled map is warped according to the equation derived for the Brownian sheet. \nFinally, the warped noise is down-scaled to the original size.\n \nThe method is tested on a number of vision problems. According to the provided evaluations, typically it produces better temporal coherence than the baseline methods.", "questions": "1. As indicated in Tab. 1, bilinear interpolation often results in quite decent quality (excl. Video SR task). Haven't the authors considered a small modification of bilinear interpolation which is able to preserve the variance of pixels, namely, replacing the commonly used weighting coefficients with their square roots? Probably, this simple modification can even further improve its performance. Also note that Eq. 5 is also essentially \"Gaussian averaging\" of subpixels, i.e. averaging using variance-preserving weighting coefficients of $1/\\sqrt{k}$ instead of $1/k$.\n1. As mentioned above, the presented method is less computationally efficient than the baselines. It would be nice to provide a quantitative evaluation of that (in)efficiency.\n1. From the manuscript, it is a bit unclear how the 9 triangulation points are warped in Fig. 1a. How exactly is the estimated optical flow $\\mathcal{T}^{-1}$ which typically has the same resolution as the image itself, applied to non-integer pixel coordinates? In other words, how is the operation $\\textrm{WARP}_{\\infty}$ from Algorithm 2 implemented?\n1. I suggest adding an explicit proof of why $\\textbf{Z}$ is a Gaussian vector (Appendix B.2). Probably, the shortest proof is the one considering the linear combinations $\\kappa^T \\textbf{Z}$ for all possible $\\kappa$, which are obviously Gaussian random variables.\n1. I recommend adding more comments on how Eq. 24 for the continuous case turned into Eq. 5 for the discrete case. Why is it valid to still rely on the first-order approximation?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "From my point of view, this paper tackles a pretty important problem for the community, since the application of image translation models to videos is a popular direction which still suffers from flickering and other artifacts. It seems to me that the Brownian sheet model and conditional Gaussian upscaling both are a very good fit for that problem as they take into account the intrinsically hierarchical nature of image resolution. The video results in the supplementary demonstrate that the amount of flickering and, vice versa, texture sticking in videos is reduced.", "weaknesses": "1. From the theoretical point of view, the presented approach relies on the warping field being a diffeomorphism, while in practice to the best of my knowledge optical flows estimated with off-the-shelf methods are rarely invertible mappings.\n1. As the authors themselves admit (Appendix E.3), \"the impact of the noise scheme is negatively correlated with the amount of constraints given to the model\", and, in particular, the method has a limited impact on latent diffusion models. However, it is obviously still useful for cascaded models.\n1. As mentioned in the paper (Sec. 6) the method is computationally less efficient than other techniques.\n1. According to Tab. 1, while the proposed method typically increases temporal coherence, at the same time it worsens frame-wise image plausibility metrics such as LPIPS or FID.\n1. None of the video samples in the supplementary except for fluid dynamics provides the results of noise handling with the method proposed by Control-A-Video model. Please note that this model more close to the presented method in spirit since it also takes input video into account while PYoCo does not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bxgUDtCuXN", "reviewer": "Reviewer_tVGv", "summary": "This paper studied the influence of initialized noise when adopting diffusion based image model to video model.\n\nA new initialization method called integral noise is propose, which is composed of a conditional upsampling, rasterization, and aggregation stage. It can maximizing the correlation between the warped and the original sample and maintain the independence of pixels in each sample.", "questions": "I am happy to see the comparison with the video processing model and the integration with DeepFloyd", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method is well illustrated in Figure 2 and is easy to understand.\nA deep and comprehensive theoretical analysis is provided in the method and supplementary material.\nThe method is evaluated on several tasks, including rerendering (SDEdit), pose-to-person, and video restoration.\nGood video illustration in supplementary material.", "weaknesses": "I guess most experiments in this paper are conducted in zero-shot way which applies the image processing model and method to video processing problems. Looking forward to seeing whether this helps when training a video model, like the PYoCo model\nConsidering video restoration (**Zeroscope Text-to-Video**) and pose-to-human (dreampose) already have good performance, I wonder the gap between image model+ integrate noise and these video models.\nAs stated in Appendix E, integrat noise does not work well in latent diffusion model. I wonder if this is applicable to other pixel-level diffusion models like the open-sourced deepfloyd from stabilityAI, whose capacity is comparable or better than stable diffusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)