Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Authors: Giannis Daras
Weili Nie
Karsten Kreis
Alex Dimakis
Morteza Mardani
Nikola Borislavov Kovachki
Arash Vahdat
Venue: NeurIPS 2024 poster (2024)
Keywords: noise warping
inverse problems
diffusion models
equivariance
Primary area: diffusion_based_models
TL;DR: We introduce a principled approach to apply diffusion models trained on images to solve video inverse problems.

Abstract: Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos as a sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only on **images** and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the underlying spatial transformations. To ensure temporal consistency, we introduce a simple post-hoc test-time guidance towards (self)-equivariant solutions. Our method allows us to deploy state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and $8\times$ video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results in the following URL: https://giannisdaras.github.io/warped_diffusion.github.io/.

Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Giannis Daras* UT Austin
Weili Nie NVIDIA
Karsten Kreis NVIDIA
Alexandros G. Dimakis UT Austin
Morteza Mardani
Nikola B. Kovachki NVIDIA
Arash Vahdat
NVIDIA
NVIDIA
Abstract
Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only images and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the under- lying spatial transformations. To ensure temporal consistency, introduce sim- ple post-hoe test-time guidance towards -quuivrqqu solutions. Our method allows to deploy state-of-the latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and 8x video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results the following URL: https:/giaamisdaara.gtttuu
diffusion, githubbboggt
Introduction
Impus Video
Diffusion models (DMs) 77
39 82 can synthe- size photorcalistic imagery 77 64 66
5
60 26]
They can be conditioned easily, through ex- plicit training also been widely used to solve inverse prob-
guidance [25] 4 and have
Warp yourr Noiee
lems [19 86 [5 16
80 84] 47 (56), in particular for image processing applications like inpainting and super-resolution
[40 72 74 66).
Warped Dilisii
How do these methods extend to video pro- cessing and solving inverse problems videos? Although video DMs are seeing rapid progress 38
(Ours)
78
29
g
general text-
to-video synthesis has not yet reached the level robustness and expressivity comparable to modern image models. Moreover, no titte-of-theaate art video generative models are publicly available (II), and most video DMs are computationally expensive. To circumvent these challenges, natural research direction is to leverage existing, powerful image generative models solve video inverse problems.
Figure I: Inpainting results for "a robot sitting on a bench". As the input video shifts smoothly, our output frames stay consistent.
TTee work was done during internship at NVIDIA.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Naively applying image DMs to videos in a frame-wise manner violates temporal consistency. Previous works alleviate the problem by fine-tuning on video data or by warping the networks'
features, using, for instance, temporal or cross-frame attention layers [8 54
13
51
61
90
92 34
[3] However, these methods are usually designed specifically for high-level text-driven editing or stylization and are typically not directly applicable to general inverse problems. Moreover, without training on diverse video data they often cannot maintain high frequency information across frames. For detailed discussion of the related works, we refer the reader to Section B in the Appendix.
The recent novel work, "How Warped Your Noise" 14, proposes noise warping to achieve temporal consistency in generated videos by changing appropriately the input noise to the diffusion model. Videos can be thought of as image frames subject to spatial transformations. An object may move according to a translation; complex and general transformations can be described by motion vectors on the pixels defined through optical flow 27 It is these transformations that define how the noise maps need to be warped and transformed. In 14], temporally consistent noise maps are given as input to the DM's denoiser, with the underlying assumption that temporally consistent inputs induce temporally consistent network outputs. In this paper, we argue that this assumption only holds true if the utilized image DM is equivariant with respect to the spatial warping transformations. However, as we show in this work, the network is not necessarily equivariant because i) the conditional expectation modeled by the DM may not be equivariant, and, ii) more importantly, a free-form neural network, as used in typical DMs, will not learn perfectly equivariant function. When the equivariance assumption is violated, the method proposed in [14] achieves poor results. This is typically the case for challenging conditional tasks (see Figure [ or when modeling complex distributions. Particularly, [1] finds that the proposed method has "limited impact on temporal coherency" when applied to latent diffusion models and that "all the noise schemes produce temporally inconsistent results".
We introduce a new framework, dubbed Warped Diffusion, for the rigorous application of image DMs to video inverse problems. We employ continuous function space perspective to DMs [52 59 28 35
that naturally allows noise warping for arbitrarily complex spatial transformations. Our method generalizes the warping scheme of 14 and does not require any auxiliary high-resolution noise maps. To achieve equivariance, we propose equivariance self- guiddnce, a novel sampling mechanism that enforces that the generated frames consistent under the warping transformation. Our inference time approach elegantly circumvents the need for additional training. This unlocks the use of cxisting large DMs in fully equivariant manner without further training, which may be prohibitive for practitioner.
We extensively validate our method on video inpainting and super-resolution, Super-resolution repre- sents situation with strong conditioning, while inpainting requires large-scale, temporally coherent synthesis of new content. Warped Diffusion outperforms previous methods quantitatively and qualita- tively, and shows reduced flickering and texture sticking artifacts. Due to our equivariance guidance, our method can also be used with lateni DMs, which is not possible with previous approaches. Virtually all existing of-te-of-the-rt rext-eo-image generation systems are indeed latent DMs, like Stable Diffusion [66]. Hence, any inverse problem solving method must be readily usable with latent DMs. In fact, all our experiments utilize the state-of-the text-to-imase -miage latent DM SDXL 60
Contributions:
(a) We propose Warped Diffusion, novel framework for applying image DMs to video inverse problems. (b) We introduce a principled scheme for noise warping, based on Gaussian processes and function space DM perspective. (c) We identify the equivariance of the DM as a critical requirement for the seamless application of image DMs to video inverse problems and propose an inference-ting -ireeneceeeeeeeeeeeee guidance method to enforce it. (d) We comprehensively test Warped Diffusion and achieve state-of-the-art video processing performance when considering the use of image DMs. Critically, Warped Diffusion can be used with any image DMs, including large-scale latent DMs.
2 Functional Video Generation
The basis of our approach, summarized in Figure 2 is to structure the generative model so that it is equivariant with respect to spatial deformations and apply these deformations successively to the input noise. Each deformation effectively warps the noise and the equivariance guarantees that each output image will be similarly warped. By using an optical flow from real video to define a sequence of such deformations, a new video can be generated. To introduce our method, we first conceptualize both im- ages and noise as functions on domain and the generator as mapping between two function spaces.
2

---

lene-lonclerion
Opeiaal
Equirarsoes
Eiput
Phocos
Farkee
Sansple
(c)
Flow
tmme per thp
Sumple
Eqpanccase
Sumpler
(4)
Cureat
Frane
Figure Visualization of Warped Diffusion applied to video super-resolution. (a) We develop a func- tion space diffusion model that super-resolves images given samples from Gaussian process (GP). To extend the image model to videos, (b) we extract warping transformations between consecutive input frames using optical flow. (c) We use the flow warp the GP sample from the previous frame. (d) To ensure temporal consistency, we introduce equivariance self-guidance in the ODE sampler.
2.1 Functional Generative Modeling and Videos
Each video frame can be seen as single image, and image discretization of vector-valued function on a rectangular domain. Consider the domain as the 2-D unit square D [0, defining an image function f D + R For each location E D, the value f(x) R represents an RGB color. We assume images have infinite resolution. To formulate a model that generates such images, must have notion of space containing all possible images. We'll use the separable Hilbert space H = (D; with pointwise formulas interpreted almost everywhere with respect to the Lebesgue measure.
We assume that there exists probability measure μr on whose support the set of photorealistic images and denote by 1) known reference probability measure on H. In our case, will be Gaussian measure on H;; for details, see Section 3.1 A generative model, or transport map, is then a mapping G H + H such that the pushforward of under G is which we denote as GşU μ. In particular, this implies that any random variable & ~ will satisfy G(E) ~ μ. For diffusion models, G' can be defined by the probability flow ODE; see Section 3.2
Given an image fo H=+1, where each subsequent function is obtained, at least partially, from the previous one by deformation. Specifically, sequence of bounded, injective maps (T, D - >>)}= exists such that Image
video with n+11 N frames is the sequence of functions fi....... fx)
Optical Flow
VxE D D OnDD Dj j= 1.
Frame Index
Pixel Location
Frame of Vision
Deformed Domain
where D, TT and assume that the sets D D, have positive Lebesgue measure. In video modeling, the sequence (T,) is usually referred to as the optical flow as it specifies how each pixel in the previous frame moves to the next frame. While the frames continuum in time, we work with a discrete set frames for simplicity. We consider / to always rep- resent fixed frame of vision and allow each T₃ to move pixels outside of this frame. Therefore (I) determines fa only on the set D n Dj which contains pixels that remain within our field of vision.
also be conceptualized
2.2 Video Generation and Equivariance
Given our notion of a video and generative model, we now describe how such a model can be used to generate new videos. Suppose we want to create a two-frame video given initial frame fo H and a deformation map D - D₁- Assume we have generative model G H + H and an initial noise image Eu H such that G(Eo) fo. From definition, the new frame of our video is

---

fi f on D1. D D1, it might seem that our generative model is unnecessary. However, proceeding this way generates blurry and unrealistic videos.
The primary issue is that, in practice, we don't have access to fo at an infinite resolution but only our grid points, need the values of fo at the points (Ex) (TI (z1)...., TT (xL)) )-x))) It's highly unlikely that EA TT' ( (Fx) for any realistic deformation.
fixed, finite set of grid points E& 1.1:-..,,2
2x) D. To determine
Thus, we must interpolate fo to (Eg), which usually leads to blurry results with standard methods. Furthermore, if D & D,, there will be regions where fi is not determined by fo and will need to be inpainted on the new visible domain. Therefore, for each frame, we must solve an interpolation and an inpainting problem: tasks for which generative models are well-suited.
Suppose we have access to the noise function Eu at infinite resolution, and its domain extends to all of R²; we discuss both in Section 3.1 We can then define the new frame in our video by applying the generative model the deformed noise: fi G(Eo D. The deformed noise function Eo Ti' gets its values from EolD for points in D n D1 and from the extension of En to I² for all other points where inpainting needed. To ensure this definition is consistent with (I). G must be equivariant with respect to Ti- Specifically, for all & supp(n) S H, must have
G(E TT (x)) (T) (x)),
VIEDnDs
Assuming 2 it follows from G(Eo) fo. that fi(x) = G(E))D)(=) = o o TT (x) for all D Dj hence the pair (Jo- fi) is valid 2 frame video according to the definition of Section 2.1 To generate a video with any number of frames, we simply iterate on this process with a given sequence of deformation maps. Enforcing 2 can be done directly by the architectural design, through training with various deformation maps, or, through guidance process; see Section 3.2
2.3 White Noise
It is common practice to train generative models assuming the reference measure is Gaussian white noise. Specifically, a draw & 1) on the grid points Eg. {z1,... ................................ .x}} C. realized as {(x²) = Xi for an i.i.d. sequence XI -V(0)) 1) for 1,..., k. However, this approach is incompatible with our goal of having the generative model perform interpolation. For most deformations T encountered in practice, none of the points in T=x(Ex) will match those in Eg. Consequently, each new evaluation (<)))(x)) will be independent of the sequence {xi)/_1- making & <(T-1(Ex) appear as a new noise realization unrelated to E(Ex). This incompatibility arises because white noise processes are distributions, not regular functions, meaning realizations are almost surely not members of H (18). [4] proposes stochastic interpolation method to address this issue (see Appendix [ for details and comparison). We generalize this idea and propose using generic Gaussian processes on HI
3 Method: Warped Diffusion
In Section [ we formulated the problem of video generation as the computation of a series of functions warped by an optical flow and proposed the use of generative model for inpainting and interpolating the warped functions. The main challenges which remain are defining a functional noise process which can be evaluated continuously and a generative model which is equivariant with respect to warping. We propose to use Gaussian processes for our functional noise and a guidance procedure within the sampling step of diffusion model to overcome these challenges.
3.1 Gaussian Processes (GPs)
A Gaussian Process (GP) 17 is probability measure on H completely specified by its mean element and covariance operator. For a mathematical introduction, see Appendix B We identify Gaussian processes with positive-definite kernel functions 2 IR.2 - IR. Recall that Egg = {.1,.... ... 3x denotes the grid points where we know the values of an image f E H. To realize a random function & 7) on these points, we sample the finite-dimensional multivariate Gaussian N(0, Q), where
Q E Rk*k is the kernel matrix Qij = x,);, for i 1,...,k k.
Once sampled, given the fixed values {(Fx), E can be evaluated at any new point xt E D by computing the conditional distribution {(x") ((E+) 165). This approach allows us to realize

---

random functional samples at infinite resolution through conditioning, thus resolving the interpolation problem. Furthermore, by ensuring the kernel K is positive definite on domain larger than D. we can consistently sample outside of D, addressing the inpainting problem described in Section 2.2
For high-resolution images when & is large, working with the matrix Q can be computationally expensive. Instead, we propose using Random Fourier Features (RFF) to sample 7, which amounts finiteddimensional projection of the function & ~ 7) that converges the limit of infinite features We can approximate samples from a GP with a squared exponential kernel with length-scale
63 87
parameter > 0 by É(x) - / Wy cos ((2,, + bÿ) for i..d. sequences wj ~ (O, 1),
2j N(0, c 12). by ~ U10, 2n) where E is the number of features. RFF allows us access to at infinite resolution on the entirety of the plane while also allowing for efficient computation.
3.2 Function Space Diffusion Models and Equivariance Self-Guidance
We will now focus on the generative model that needs to be equivariant to the noise transformations. Specifically, in this section, i) we introduce function space diffusion models, ii) we prove that if every prediction of the diffusion model is equivariant then the whole diffusion model sampling chain is equivariant to the underlying spatial transformations, and, iii) we describe equivariance self- gelf-guidance,, our sampling technique for enforcing the equivariance assumption.
For ease of notation, we will present everything for the case of unconditional video generation. However, our method seamlessly incorporates any addition conditioning information that may be available. If .o..... G. E R' is sequence of known conditioning vectors then these can simply be passed into conditional score model at the appropriate frame without any other change to our method; see A Conditioning vectors could be, for example, low resolutions versions of video or an original video with regions masked. In Section 4
we focus on such conditional tasks.
Function Space Diffusion Models. Typically, diffusion models are trained with white noise. As explained in Section 2.3 a principled continuous evaluation of the noise requires functional process. We briefly describe diffusion models in the context of sampling using the Gaussian processes of Section 3.1 We show in Section 4.1 (Table that model trained with white noise can be fine-tuned to GP noise without any loss in performance.
While it is possible to formulate diffusion models the infinite-dimensional space H e.g. 52 will proceed in the finite-dimensional case for ease of exposition. In particular, we will define the forward and backward process a flow scalar function evaluated on the grid Eg and recall that Q is the kernel matrix on Eg.
vector R thinking of the entries as the values of
We consider forward processes of the form,
du; (2o(t) 1/2 dw, u(0) up ~ μ
where W. is standard Wiener process on R and is scalar-valued, once differentiable function. This process results in conditional distributions p(u, (wo) = N N(uo, ( see [46). Let p(u,,) I) denote the density of un induced by 33). Then the following backward in time ODE,
du, dt
(1) log t(u.,
started at u(r) distributed according to 33 has the same marginal distributions p(u,, t) as 3 on the interval [0, 7]: see [46]. Approximating N (uo, (²) by [0, (T))))). we may then define the generative model G by the mapping u(r) + u(0) with reference measure 7 = NN o² (T))).
Solving (4) requires knowledge of the score V. log pl (1, t). Instead of learning the score, we opt for directly learning the weighted score QV, a log p(u,,t). t). This design choice leads to faster sampling since we do not need perform any expensive matrix multiplication with Q at inference time.
generalized version of Tweedie's 's formula (for proof see Appendix A.2 implies:
E6 Eluo [u]]
QV log t)
(5)
(2(()
We approximate Ejugl u.] with neural network hy by minimizing the denoising objective:
E E,~U(0 Eno Bu N(n, (1e, un! 2
(6)
5

---

Having a minimizer he of 6 gives us access to the weighted score QV log p(u,.t) t) via ( We may then obtain an approximate solution the map u(T) + u(0) by discretizing 44) in time. We consider Fuler scheme updates given by
(()) o(t)
- = Ue At
ho (ue, t) - me).
started with u, ~ (0, Q²())) for some time step At > O.
Equivariance for the Probability Flow ODE, Since the diffusion model works with discrete inputs, we need to introduce a discretization of 2 for the network. For a deformation Ti, we define equivariance as
he (wt o ()) ())
88)
which is obtained from composing both sides of (2) with T.. Note that (8) is valid only for pixels the network output. The input to the network on the l.h.s. is computed with RFFs without any interpolation. Given this discrete equivariance is satisfied for every prediction of the network, it is straightforward to show that the whole diffusion model sampling chain will be equivariant. Indeed, the whole approximation to u(T) u(O) is equivariant by the linearity of composition for a full derivation, see Appendix AA.3
which stay within frame and we compute the lh.ss with bilinear interpolation
The condition (8) is rarely satisfied for deformations Ti arising in practical settings. This is because either the conditional expectation E/uolut [u.] is not equivariant with respect to T or the neural network approximation has not fully captured it. If the underlying equivariance assumption breaks, methods that rely solely on noise warping for temporal consistency, e.g. [14], will perform poorly. This is evident in challenging conditional tasks (see Figure
Equivariance Self-Guidanee
A potential solution is to directly train the network by adding as a regularizer. However, this requires large amounts of video data from which to extract optical flows. Furthermore, by satisfying (8 over large class of Ti (s), the network may become less apt satisfying (6) and lose its generative abilities. Therefore, we opt for guiding the model towards equivariant solutions at inference time.
We first sample noise u and generate the first frame following keeping the outputs of the network at each time step (ha(u, (ho(uu(),))) .t). To generative the next frame, we warp our noise (L) = u² (") o 'T with RFFs and again follow 7 but this time using (8) as guidance. In particular, we take gradient steps in the direction of the loss function hh (2) oi ho uu, (o) I. computed on the pixels that stay within frame. All frames can be generated by iterating this procedure as summarized in Algorithm [] (and visualized in Figures 2 ( which also shows how use conditioning information. Guidance is typically used to solve inverse problems with diffusion models (e.g. see [5)., but here the guidance is applied to align the model with its own past predictions. We emphasize that to compute the composition with Tj above, we use bilinear interpolation on the network outputs but we never need to interpolate the network inputs since we can compute the warping via RFFs. Furthermore, since we are matching interpolated outputs to ones that are not interpolated, our output images remain sharp. This in contrast to directly using a discrete version of (2) which would suggest that we match network outputs to interpolated images, producing blurry results.
4 Experimental Results
For all our experiments, use Stable Diffusion XL 60 (SDXL) as our base image diffusion model. We start by finetuning SDXL. on conditional tasks. We choose super-resolution and inpainting as the tasks of interest since they are both commonly used the inverse problems literature and they rep- resent two distinct scenarios: in super-resolution, the input condition is strong and in inpainting, the model needs to generate new content. For super-resolution, we choose a downsampling factor of 8. For inpainting, we create masks of different shapes at random, following the work of [57]. During the fine- tuning, we train the model to predict the uncorrupted image given the following inputs: i) the encoding of the noised image. ii) the noise level, and, iii) the encoding of the corrupted (downsampled/maskec) image. To condition on the corrupted observation, we concatenate the measurements across the channel dimension. We train models with and without correlated noise on the COYO dataset [12] for 100k steps. We show realizations of independent and correlated noise in Figure Additional imple- mentation details are in Section F2 including the parameters for the GP introduced in Section 3.]

---

Algorithm/ 1 Warped Diffusion - Temporal Consistency with Equivariance Self Guidance
Require: Conditioning vectors {o:] Step At, Time т, Schedule a (t), Model hgg Guidance Strength A.
1: (1 f- tttpeee__ppccclllloloooo
2: ur ~ GP using RFFs in Section 3.1
Fresh noisc sample for the first frame b Sample first frame
3: Compute trajectory {24
(0) using
4:
for <- 1 to do
5:
u² + u7
(5)
11
"Tg using RFFs
b Warp noise from previous frame
6:
f+r
7:
while > do
u x- (1) D -1 (holue (1) ,c, -u
D Take Euler step
9:
at - [ (1) ,4,0)) TJ holu (1))))) .,,,,,4,0) ,,,
Compute warping Equivariance self guidance
10:
u u t u Vcc
Vuee
11:
t t- At
12:
end while
13:
end for
14:
return video
(3) {uo 1
4.1 Training with correlated noise
The first step is to assess the quality of the trained models. To do so, we take images from a test split of the COYO dataset, we corrupt them (either by masking or downsampling) and we measure the conditional performance of the trained models. We use a diverse set of metrics that are commonly used in the inverse problems literature: CLIP Text Score 62), CLIP Image Score [2]. SSIM [85).
LPIPS 191, MSE, Inception Score [76] and FID (37. The first five metries measure point-wise restoration performance. Inception Score measures the quality of the generated distribution (without an explicit reference distribution). Finally, FID measures restoration performance in a distributional sense, i.e. it measures how close is the distribution after restoration to the ground truth distribution.
We report our results for the super-resolution and inpainting models in Table The main finding is that finetuning with correlated noise does not compromise performance, i.c. SDXL models finetuned with correlated noise perform on par with SDXL models that are trained with independent noise. Particu- larly for inpainting, the GP models slightly outperform models trained with independent noise across all metrics. We provide qualitative results for our all models in Figure ! and in Appendix Figures 7
We remark that the advantages of using an initial distribution other than white noise have been explored in prior work 20 6 42 Our new finding is that model initially trained with white noise can be easily fine-tuned to work with correlated noise. To the best of our knowledge, ours is the first work that shows that Stable Diffusion XL can be fine-tuned to work with correlated noise.
We underline that prior to fine-tuning Stable Diffusion XL produces unrealistic images when the sampling chain is initialized with correlated noise. Our experiments show that post-finetuning, the model can handle spatially correlated noise in the input without compromising performance. Our GP Warping mechanism requires models that handle correlated noise. Hence, these fine-tunings essential for the rest the paper.
Table I: Single-framme evaluation of super-resolution and inpainting models.
Model
FID |
Inception f
CLIP T
CLIP Img
SSIM
LPIPS 1
MSE 1
Super-resolution GP
37,514
11.917
0.272:na22
t.955
0.770 106
0.253+tamm6
0.004
Super-resolution Indep.
40.843
11.679
0.27 (.271+to +004)
0.957=aa17
0.785
0.242:1ams
0.004 +0.00
Inpainting
58.727
11.769
0.276:na2
0.929=aum60
0.798=0.134
0.181
0.056=a.084
Inpainting Indep.
61,380
11.707
0.275+ta04
0.913 ++0.000
0.778++0.16
0.118++a144
0.057++.0m6
4.2
Noise Warping and Equivariance Self Guidance
In the previous experiments, we measured the restoration performance of the trained models for a single image and we established that models trained with correlated noise perform par (or even outperform) models trained with independent noise. The next step is to measure the temporal behavior of the models, i.e. how well they work for videos.
7

---

Noise Warping baselines. As explained in Section I to apply image diffusion models to videos, we need to transform the noise as we move from one frame to the next. We consider the following noise-warping baselines that were used in [4] Fixed noise uses the same noise across all the frames. Resample noise samples a new noise for each new frame. Nearest Neighbor uses the noise of the nearest location in the grid to evaluate the noise at the location that is not on the regular grid Egg- Bilinear Interpolation interpolates the values of the noise bilinearly in the neighboring locations that lie the grid. How I Warped Your Noise 114 is the state-of-the-ar method for solving temporally correlated inverse problems with image diffusion models. It warps the noise by using auxiliary rightresolution noise maps (see intro, related work section, and Section C) Our GP Noise Warping warps the input noise by resampling the Gaussian process in the mapped locations. We note that the Fixed Noise, Resample Noise, and Nearest Neighbor noise warping methods can be applied to models that are trained with either independent noise or correlated noise coming from a GP. For all the experiments, also include our proposed method, Warped Diffusion that uses GP Noise Warping and Equivariance Self-Guidance (see Algorithm foor a reference implementation).
Video Evaluation Metries. We follow the evaluation methodology of the "How I Warped Your Noise" paper [4]. Specifically, we want to measure two different aspects of our method: i) average restoration performance across frames, ii) temporal consistency. For i), we measure the average of all the previously reported metrics (FID, Inception, CLIP Image/Text score, SSIM, LPIPS and MSE) across the frames. For ii), measure the self-warping error, i.c. how consistent are the model's s predictions across time. The warping error can be computed in either pixel or latent space and also with respect to the first generated frame or the previously generated frame, totaling 4 warping errors.
To warm up, we start with videos that are synthetically generated by 2-D shifting of a single image, as in Figure [ To further simplify the setup, we consider the easy case of shifting the current frame by an integer amount of pixels with each new frame. For 2-D translations by an integer amount of pixels, the Nearest Neighbor, Bilinear Interpolation, How I Warped Your Noise and GP Noise Warping methods they become essentially the same since we always evaluate the noise distribution on points in the grid Eg. Hence, the only difference is whether we apply these methods to white noise to GPs.
Figure (Row 2) shows that the How I Warped Your Noise baseline produces temporally inconsistent results as we shift the masked input image. Even though all the inpaintings are of high quality, the baseline results are temporally inconsistent. Instead, our Warped Diffusion method produces temporally consistent results since it enforces equivariance by design. Since the How I Warped Your Noise warping mechanism and GP coincide here, the benefit strietly comes from enforcing the equivariance property. In fact, one could get the same results for the How I Warped Your Noise method by penalizing for equivariance at inference time.
We present quantitative results regarding temporal consistency in Figure 3 (and additional results in Figure4fin in the Appendix). As shown in the Figure, the fixed noise and the resample noise baselines perform the worst w.r.t. the temporal consistency both in latent and pixel space. The warping error of the Resample baseline is almost constant across frames as expected, while the warping error of the Fixed Noise increases with time. Both the How I Warped Your Noise method and our GP warping framework significantly improve the baselines. Yet, they still have significant temporal inconsistencies as evidenced by the results in Figure perform on par on this task since they are essentially the same when it comes to integer shifts: the only difference is that GP Noise Warping is applied to correlated noise coming from GP. The remaining temporal errors are not an artifact of the noise warping mechanism but they are due to the fact that the model itself is not equivariant w.r.t. the underlying transformation. The warping errors essentially disappear when we apply Equivariance Self Guidance. As shown in Figure 3, our method, Warped Diffusion, achieves almost 0 warping error (Ic-4 mean pixel error with respect to the first frame to be precise) since it is enforcing equivariance by design.
the supplemental videos. The two methods
The only remaining question is whether Warped Diffusion maintains good restoration performance. To answer this, we measure mean restoration performance across frames for the aforementioned metrics. We report our results in Table 2 frame. As shown, Warped Diffusion maintains high performance across all the considered metrics while being significantly superior in terms of temporal consistency. The conclusion is that all the other noise warping baselines, including the previous state-of-the-as How I Warped Your Noise paper [4), perform poorly in terms of temporal consistency since they rely on the assumption that the network is cquivariant. Even for simple temporal correlations such as integer movement in the 2-D space, this
including the mean warping error with respect to the first
8

---

Inpainting
Inpainting
Salf warsing latent orter wre fist frattrame
Rsed
0.150
Rasample (ap)
0.6
Resample Unddepl
0.125
GP Nose Warping
Ours (gp)
warped your (Indep)
Floe (gp)
0.015
Fleed (hheep)
Safrwerping warping
0.2
Resampls (gp)
0.000
(iddep)
0.1
Warping
0.025
top)
Frow inarped your nobee (indep)
10
12
14
10
12
frame Index
Frame Index
(a) warping error w.r.t first frame in latent space.
(b) Serf-ng error w.r.. first frame in pixel space.
Figure 3: Self-warping error w.r.t. first frame for the inpainting task as we shift the input frame.
assumption is false for the challenging inpainting task. Warped Diffusion is the only method that achieves temporal consistency while it still manages to maintain high reconstruction performance.
Table 2: Mean-frame evaluation of inpainting models for the translation task.
Method
Warping Eir
FID
Inception f CLIP Txt 1 CLIP Img
SSIM
LPIPS 1 MSE 1
Fixed (gp)
=.129:002:
60.853+2.98
12.421#4766
0.924 0.280=
0.182 0.800
0.060=u00:
Fixed (indep)
0.080=00:4
10.301 67.021
0.275#0002
0,919
0.195 0.780=
0.059=a002
Resample (indep)
0.101
71.078
11.740a04:5
0.277
0,921=0m
0.781=nmx
0.196
0.061=000
Resample (gp)
0.141=000s
60.029
11.318+0410
0.277=0002
0.925=a0m3
0.806-a005
0.182
0.056:0003
How I Warped (indep)
0.046:0007
68.701 222918
10.877a6.322
0.276
0.910:ams
0.781=000:
0.197
0.067
GP Warning
59.897-13718
11.727.a..75
0.277..00
0,924.nm:
0.803_000
0.057.0000
Warped Diffusion (Ours)
0.001
61.249
11.80200477
0.276
0,917
0.779 88
0.058=000
We finally remark that our sampling algorithm enforces equivariance in the latent space. Yet, the
warping errors are negligible in the pixel space as well. Our finding is that improving latent space equivariance translates to improvements in pixel space equivariance, The authors of 14 also find that "the VAE decoder is translationally equivariant in discrete way".
4.3 Effect of Sampling Guidance for more general transformations
We proceed evaluate our method realistic videos. We measure performance 600 captioned videos from the FETV (55] dataset. Since baseline inpainting methods fail even for very simple temporal transformations,
focus 8x super-resolution for comparisons FETV.
For our video results, we could not provide comparisons with the How Warped Your Noise paper. At the time of this writing, there was no available reference implementation the authors by direct communication. work that their proposed method has "limited impact temporal coherency" when applied to latent models and that "all the noise schemes produce temporally inconsistent results" [14]. Once again, we attribute this the non-equivariance of the denoiser, which mitigate with guidance algorithm.
confirmed with any case, the authors acknowledge as a limitation of their
We proceed evaluate our method and the baselines with respect to temporal consistency and mean restoration performance across frames, as we did for our inpainting experiments. We present our results in Table additional results in Figures 5 of the Appendix and in the following URL, as videos: https::/ggannisdaae www.uu..tttubu...g.. io/ fo/warped diffusion, lttfiuiiiiongttb.iiio.oo As shown in Table 3 there is trade-off between temporal consistency and restoration performance. Methods that perform better in terms of temporal consistency often have significantly worse performance across the other metrics. Our Warped Diffusion achieves a sweet spot: it has the lowest warping error by a large margin and still maintains competitive performance across all the other metrics. On the contrary, methods that are based solely on noise warping, such as GP Warping and the simple interpolation methods, lead to significant performance deterioration for small improvement in temporal consistency.
Noise Warping Speed. We measure the time needed for a single noise warping. Our GP Warping mechanism takes 39ms per frame Wall Clock time, to produce the warping 1024 1024 resolution. This is 16x faster than the reported 629ms number [4]. If we use batch parallelization, our method generates 1000 noise warpings just 46ms (at the expense of extra memory).
No Warping? natural question is whether we can omit completely the noise warping scheme since equivariance is foreed at inference time. We ran some preliminary experiments for super-

---

Table 3: Mcan-framer evaluation of super-resolution models for real videos.
Method
Warping Ear 4
FID
Inception
CLIP Txt
CLIP Img 1
SSIM
LPIPS
MSE 4
Fixed (indep)
0.940-1o.:1
48.764
8,746
0.227
0.948
0.7
0.005 10.0
Resample (indep)
(Indep)
0,93411334
47.550
8.879=1397
0.229.1mm2
0.9481o01
0.708
0.183-001
0.005
Nearest (indep)
67.078 1
8,608
1.w
0.228 Lomm
0.943
0.683
0,227
0.007:1..00
Bílinear (indep)
0.990
66.330-t4..
wu
8.832-1000
0.228+bax
0.9424o0m
0.68440.09
0.216:ta0s
0.008+0000
Fixed (gp)
1.006+tu wat
54.058 0015844400
8.045-1205
0.222-Hoam
0.954++oom
0.666+
0019--
0.007++0.00
Resample (gp)
0,974
54,778=3322
9.47
0,225-baa4
0.954
0.61
0.209=a000
0.006
Nearest (gP)
0.975
19.743420.825
8,896=14462
0.2241:.002
0:939220004
0.637:b..15
0,243=2004
0.009::0:00
Bilineur (gp)
0.953-10.390
11.866
8.565
1.577
0.228.Lo90
0.942.Losss
0.635 Loose
0,247:1a04
0.009.1..00
GP Warping
0.812-4-0.9
78.763-4-1.995
8.291-1
0.225-+amm:
0.653-Hoom
0.008-
Warped Diffusion (Ours)
0.649
58, 189
8.882=17UM
1.714
0.235
0.943+t000
0.654+bo224
0,221=mo04
0.008
resolution real-videos and we found that omitting the warping significantly deteriorates the results when the number of sampling steps is low. We found that increasing the number of sampling steps makes the effect of the initial noise warping less significant, at the cost of increased sampling time.
5 Limitations
Our method has several limitations. First, the guidance term increases the sampling time, as detailed in the Appendix, Section F33 For reference, processing a 2-second video takes roughly minutes on single A-100 GPU. Second, even though in our experiments we observed monotonic relation between the warping error latent space and warping error pixel space, it is possible that for some transformations the decoder of a Latent Diffusion Model might not be equívariant. We noticed that this is common failure for text rendering, e.g. in this latent video the model seems to be equívariant, but in the pixel video it is not. Third, the success of our method depends on the quality of the flow estimation inconsistent flow estimation between frames will lead to flickering artifacts. For real videos, there might be occlusions and the estimation of the flow map can be noisy. We observed that such cases our method fails, especially for challenging tasks such as video inpainting. The correlations obtained by following the optical flow field obtained from real videos might lead to distribution shift compared to the training distribution. For such extreme deformations, our method produces correlation artifacts. This has been observed in prior work (see this video), but it also appears in our setting (c.g. see this video). Finally, our method cannot work in a zero-shot manner since it requires model that is trained with correlated noise.
Conclusions
Warped Diffusion is a novel framework for solving temporally correlated inverse problems with image diffusion models. leverages noise warping scheme based on Gaussian processes to propagate noise maps and it ensures equivariant generation through an efficient equivariance self eeff-guidance technique. We extensively validated Warped Diffusion on temporally coherent inpainting and superresolution, where our approach outperforms relevant baselines both quantitatively and qualitatively. Importantly, in contrast to previous work [14], our method can be applied seamlessly also to latent diffusion models, including state-of-the-ee the-art text-to-imant to-image models like SDXL [60].
Acknowledgements
This research has been partially supported by NSF Grants AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844 and research gifts by Western Digital, Amazon, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco and the Stanly P. Finch Centennial Professorship in Engineering. Giannis Daras has been partially supported by the Onassis Fellowship (Scholarship ID: F ZS 012-1/20222-0223) the Bodossaki Fellowship and the Leventis Fellowship.
References
[1] Asad Aali, Marius Arvinte Sidharth Kumar, and Jonathan I Tamir. Solving inverse problems with score-bee based generative priors learned from noisy data. arXiv preprint arXiv:2305.01 166.2000.0166661000 2023.
10

---

[2] Asad Aali. Giannis Daras. Brett Levac Sidharth Kumar, Alexandros G Dimakis, and Jonathan I Tamir. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained corrupted data, arXiv preprint arXiv:2403.08722 2024.
[3] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilisticc models. arXiv preprint arXiv:2205. 15019, 2022.
[4] Nachman Aronszajn. Theory of reproducing kemels.. Transactions of the American Mathematical Society,
68, 1950.
[5] Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song. Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu, ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:221..03324 2022.
[6] Arpit Bansal, Eitan Borgnia, Mong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noisc. arXiv preprint arXiv:2208. 09390, 2022.
[7] Omer Bar Tal, Hila Chefer, Omer Tov, Charles Hermann, Roni Paiss, Shiran Zada, Ariel Ephrat. Junhwa Hur, Guanghui Liu, Amit Raj. Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang. Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-time diffusion model for video generation. arXiv preprint ar 24v: /2945, 2024.
[8] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Volcti, Adam Letts, Varun Jampani, and Robin Rombach. Stable vidco diffusion: Scaling latent vidco diffusion models to large datasets. arXiv preprint ar arXiv:2311.15127, 2023.
[9] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High resolutino video synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[10] Valentin De Bortoli, James Thornton. Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In Advances Neural Information Processing Systems, 2021.
[11] Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[12] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee. Woonhyuk Back, and Saehoon Kim. Coyo-700m: Image-text pair dataset. lattps://github.. com wwtp:///wkkkabraaaaaooooooaaaaae
2022.
[13] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 23206-23217, 2023.
[14] Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C Azevedo. How i warped your noise: a temporally noise prior for diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[15] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems, arXiv preprint arXiv:2209. /4687, 2022.
[16] Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint (XXxv:22066009411 2022.
[17] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Diffusson steps, twists, and turns for molecular docking. arXiv preprint 2r10012210001777, 2022.
[18] G. Da Prato and J. Zabezyk. Stochastic Equations in Infinite Dimensions. Encyclopedia of Mathematics and its Applications. Cambridge University Press, 2014.
[19] Giannis Daras, Hyungjin Chung, Chich-Hsin Lai, Yuki Mitsufuji, Peyman Milanfar, Alexandros G. Dimakis, Chul Ye, and Mauricio Delbracio. A survey on diffusion models for inverse problems. 2024.
[20] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar, Soft diffusion: Score matching for general corruptions. arXiv preprint 220:22209000442.. 2022.
11

---

[21] Giannis Daras and Alex Dimakis. Solving inverse problems with ambient diffusion. In NewrlPS 2023 Workshop on Deep Learning and Inverse Problems, 2023.
[22] Giannis Daras, Alexandros G Dimakis, and Constantinos Daskalakis. Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data. arXiv preprint arXiv:2404. 10177, 2024.
[23] Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36, 2024.
[24] Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303. 2023.
[25] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
[26] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, Dustin Podell, Tim Dockhom, Zion English, Kyle Lacey. Alex Goodwin, Yannik Marck, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03200 2024.
[27] Denis Fortun, Patrick Bouthemy, and Charles Kervrann. Optical flow modeling and computation: A survey. Computer Vision and Image Understanding, 134:1-21, 2015.
[28] Giulio Franzese, Giulio Corallo, Simone Rossi, Markus Heinonen, Maurizio Filippone. and Pietro Michiardi. Continuous-time functional diffusion processes. Advances in Neural Information Processing Systems, 36, 2024.
[29] Songwei Ge, Seungjun Nah Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[30] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311. 2023.
[31] Ramesh Girish and Gopal Krishna. A survey on video diffusion models. arXiv preprint arXiv:2310.1066 2023.
[32] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and- play priors. arXiv preprint arXiv:2206.094: 09012 2022.
[33] Yuwei Guo. Ceyuan Yang. Anyi Rao. Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933, 2023.
[34] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations (ICLR), 2024.
[35] Paul Hagemann. Sophie Mildenherger, Lars Ruthotto, Gahriele Steidl, and Nicole Tianjiao Yang. Multi- level diffusion: Infinite dimensional score-based diffusion models for image generation. arXiv preprint arXiv:2303.04722 2023.
[36] Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. arXiv preprint arXiv:23/2.02133 2023.
[37] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[38] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition vidco generation with diffusion models. arXiv preprint arXiv:2210.02300 2022.
[39] Jonathan Ho Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
12

---

[40] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint vrXiv:2106.1528. 2021.
[41] Jonathan Ho and Tim Salimans. Classifier-fiee free diftusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstrean Applications, 2021.
[42] Emicl Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arXiv:2209005557, 2022.
[43] Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning (ICML), 2022.
[44] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Syxtems, 114499999444,, 2021.
[45] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. Advances in Neural Information Processing Systems, 2022.
[46] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206. 00364, 2022.
[47] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23660 2022.
[48] Bahjat Kawar, Noam Elata, Tomer Michacli, and Michael Elad. Gsure-based diffusion model training with corrupted data, arXiv preprint arXiv:2305. 2023.
[49] Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions. arXiv preprint arXiv:22 22.::22//20000 000886 2022.
[50] Gavin Kerrigan, Giosue Migliorini, and Padhraic Smyth. Functional flow matching. arXiv preprint arXiv:2305.77209, 2023.
[51] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero. Text-to-image diffusion models are zero-shot video generators. In Proceedings of the TEEF/CVF International Conference on Computer Vision pages 15954-15964, 2023.
[52] Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.0744: 2023.
[53] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie. and Anima Anandku- mar. 1ss:: tmageeto-image schrödinger bridge. arXiv preprint arXiv:2302.05872, 2023.
[54] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2pt Video editing with cross- attention control. arXiv preprint arXiv:2303.04761 2023.
[55] Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-vidoo generation. arXiv preprint arXiv: 2311.01813, 2023.
[56] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference Learning Representations. 2023.
[57] Suraj Patil. Sdxl inpainting model, 2024, Accessed: 2024-05-21.
[58] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[59] Jakiw Pidstrigach. Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimennionaal diffuusion models. arXiv preprint arXiv: 2023.
[60] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann. Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusson models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations (ICLR), 2024.
13

---

[61] Chenyang Qi. Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IELE/CVF International Conference on Computer Vision, pages 15932-15942, 2023.
[62] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[63] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[64] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 2022.
[65] C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning series. MIT Press, 2005.
[66] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. arXiv preprint vrXiv:2112.10755 2021.
[67] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[68] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI). 2015.
[69] Claudio Rota, Marco Buzzelli, and Joost van de Weijer. Enhancing perceptual quality in video super- resourresoltion through temporally-comee consistent detail synthesis using diffusion models. arXiv preprint arXiv:2311.1500 2023.
[70] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models, Advances in Neural Information Processing Systems, 36, 2024.
[71] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Flect, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[72] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2111.05822 2021.
[73] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic lext-to-image diffusion models with deep language understanding. arXiv preprint arXiv: 1488,, 2022.
[74] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07633 2021.
[75] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on paltern analysis and machine intelligence, 45(4):473-4424 2022.
[76] Tim Salimans, lan Goodfellow. Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[77] Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Conditional simulation using diffusion schrödinger bridges. arXiv preprini arXiv:2202. 1t..22222..34600 13460, 2022.
[78] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Videss Text-to- Video Generation without Text-Video Data. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[79] Jascha Sohl- -ill-Dicktten, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
14

---

[80] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guite diffusion models for inverse problems. In International Conference on Learning Representations, 2022.
[81] Jiaming Song, Arash Vahdat, Morteza Mardami, and Jan Kautz. Pseudoinverse-guiddd diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
[82] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
183] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[84] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.0044 2022.
[85] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-6) 61160000622200 2004.
[86] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint arXiv:2112.02473 2021.
[87] James Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Mare Deisenroth. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning, pages 10292-10302. PMLR, 2020.
[88] Jay Zhangjic Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: Onc-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 7623-7633, 2023.
[89] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geomet- ric diffusion model for molecular conformation generation. In International Conference on Learning Representations (ICLR), 2022.
[90] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video: Zero-shot text-guided video-to-vides translation. In SIGGRAPH Asia 2023 Conference Papers, pages 1- 2023.
[91] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[92] Zicheng Zhang. Bonan Li, Xuecheng Nie, Congying Han. Tiande Guo, and Luoqi Liu. Towards consistent vidco editing with text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[93] Shangchen Zhou et al. Flow-guided diffusion for video inpainting, arXiv preprint
2023.
A Theoretical Results
A.1 Convolutions and Equivariance
To better understand ( we consider the following example. We will assume that E IR2 R is scalar-valued field (grayscale image) defined on the whole plane. Furthermore, we let G be given by a continuous convolution and 7 be a translation. In particular, for all x E TR?
/
G(f)(z)
<( - k(z-v)((x) dy.
(x)
for some compactly supported kernel k R22 + R and a direction E R²a We then have
G( (()) = [ <(λ v) a) dy = I << (((x=a)--00 ((===))-))(()) dy ((T
by the change of variables formula. This shows that is equivariant all translations. This is a well-known property of the convolution and, in particular, it shows that convolutional neural networks
15

---

are translation equivariant, noting that pointwise non- con-lineristies will preserve this property. This example shows that a model can be equivariant with respect to certain deformations by architectural design. However, in realstic video modeling, optical flows are not know explicitly and can only be approximated numerically.
therefore natural instead build-in approximate equivariance into model instead of enforcing it directly in the architecture. Our guidance procedure in Section 3.2 is an example such an approximate form of equivariance.
A.2 Tweedie's Formula
In Section 3.2, we show a diffusion model can be trained and sampled from using Gaussian pro- cess noise instead of white noise. Our result depends on the following lemma which is a simple generalization of Tweedic's s formula.
Lemma A.1. Let x be a random variable with positive density Pa (RRRR). Let a > 0 and ~0, for some positive definite matrix Q E B.xx and assume that & z. Define the random variable
== + oz and let Py E (R*) be the density of y. It holds that
1 02 Q (E[xly] v)-
Vy log
Proof. First note that by the chain rule,
V. / ply. x) dr (1) R'
lo = (v) V VuPp(u) =
where p(y, denotes the joint density of (y) Let ply ar) denote the Gaussian density of the conditional y|a Since Pz (RR*..
p(y.z) dr / (x) dz.
Vy get
Therefore, by the chain rule,
/ plul=)p= (x) V log p(yl=) dr. JR
V (u) Py
Since p(ylz) is the density of N(x R² Q), direct calculations shows that
V. logg((/))) -1 - w).
Furthermore, Bayes' theorem implies
p(ulr)pa = p(>lv)p,(())
Therefore,
1 0² Rd
/ Q Q v)p(zl)) dr -====010 Q-((v( g2
V y log
as desired.
A.3 Flow Equivariance
In Section 3.2 we claim that the score network hg is equivarient with respect to a deformation
T-1 then the Euler scheme approximation of the map u(r) + u(0) is equivarient with respect to
T It is easy to see that this results holds so long as it holds for the single step u, H 4 defined (8) written
by We will assume that he safisfies
ho ho(ue T (,,)) t) ho (w4, (4, T
We make sense of this equation by using RFF to define as a function on the plane and similarly bilinear interpolation define ho(ue, t) as function. follows by linearity of composition that
o Uq o - (t)) (he t) o 1
(t)
i(1) (ho(us o TT - 2 1
44 o T
At
a(t
which is the requisite equivariance of the map ut Ut-Ai.
16

---

B
Gaussian Processes
probability measure 1 on called Gaussian if there exists an element m E H and a self self-adjoint, non-negative, trace-class operator Q H - H such that, for all h, h' E H,
(h, m) = () dn(s), (Qh, = (0.5-m)(i
m) dn(f):
where (,,-) denotes the inner product on H. The element m is called the mean while the operator Q called the covariance. It is immediate from this definition that white noise is not included since the identity operator is not trace-class on any infinite dimensional space. This definition ensures that any realization of a random variable elements of И is a subset of the real line, is often called a Gaussian process, We continue to use this terminology even when D is subset of a higher dimensional space i.e. IR2 but remark that the nomenclature Gaussian random field is sometimes preferred.
1) almost surely an element of H. When the domain D of the
Since working separable space, each such field H has associated to it unique reproducing kernel Hilbert space 18 Theorem 2.9] which is associated to unique positive definite kernel [4]. In particular, there exists positive definite function D D R for which Q is its associated integral operator. It follows that a Gaussian process can be uniquely identified with a positive definite kernel. Sampling and conditioning this process can then be accomplished via the kernel matrix.
To make this explicit, suppose that = { Zn) C D and Y {y1,...,, 9mm) C are two sets of points in D. We will slightly abuse notation and write
e(x,)s, == u().,1)),
i= i.....,, and j 1,...,m
for the kernel matrix between X and Y and similarly Q(Y, X), Q(X, Q),,). Suppose that Ç 17 is a random variable from the Gaussian process with kernel and mean zero. To sample realization of on the points X, we sample the finite dimensional Gaussian N No ()) This can be written as
{(X) Q(X,
where Z ~ NN L.). Suppose that the points in Y distinct from those in X and want to sample & on Y given the realization {(X). This can be done by conditioning (65)
~ -(7))EE))
Q(Y, -0XX))0XXXXXXXXXXx0x0000
While the above formulas fully characterize sampling ç. working with them can be computationally burdensome. It is therefore of interest to consider a different viewpoint on Gaussian processes, in particular, through the Karhunen- Larve expansion. The spectral theorem implies that Q possesses a full set of eigenfunctions 2, rjoo for j = 1,2... ... with some decaying sequence of eigenvalues A 2 d2 2... .... The random variable & (0, Q) can be written as
20
{ = Σ V #=1
where X, /i 1) is an i.i.d. sequence and the right hand side converges almost surely in the norm H 18). By truncating this sum to a finite number of terms, computing realizations of { becomes much more computationally manageable. This inspires the random features approach to Gaussian processes which is the basis of our computational method; for precise details, see
65 63 87
C Brownian Bridge Interpolation
We show in Section 2.3 that white noise process is not compatible with the idea of using a generative model to interpolate deformed functions. A potential way of dealing with this issue is to treat the original realizations of the white noise {(Ex) the fixed nodal points of function & and obtain the rest of the values via interpolation. It is shown in [14] that common forms interpolation yield conditional distribution E(T (Ex))]{(EE that is too dissimilar from the training distribution AO 1x) and thus the generative model produces blurry disfigured images.
17

---

Therefore [14 proposes a stochastic interpolation method which has the property that, for new point x² & Eg, the distribution of {(x*) marginalized over the joint distribution (
follows (V 1). This is most easily seen in one spatial dimension with k 2 points. Suppose that D = [0. 1] and let a, 6 ~ (00, be two independent random variables. Consider a Gaussian process on D with kernel function k(,,, 3)
x y| and suppose that & distributed according to this GP conditioned on {(0) = a and {(1) = b. A straightforward calculation shows that, for any
x E (0,1). 1),
{(x*) = (1 11xxx"p =²)a- + x: + 2x x=)>
for = N(0. 1) independent of (a, This simply the Brownian bridge connecting a to b. Remarkably, the marginal distribution of [(x") ) over the joint (E(=*), b) is (0, independently of x². However, the conditional distribution is
E(x" )a, = -N((11 x²) x²)a + x² 2x -1)))) x²))
which not (/ 1) for all E (0, 1). In 14 Section 2.21, proposed that such Brownian bridges are used between any two pair of pixels. yielding stochastic interpolation method given by sequence of such independent GPs. However, from the point of view of using a generative model that is pre-trained on (0, I&) ). it is not of interest that the marginal distribution of {(=*) is (V 1) but rather that the conditional [(x=) )la, b is (0, 1). As we have seen, this is not the case for the method of [14) and, in fact, it will only ever be the case for white noise processes discussed in Section 2.3 Therefore, no matter what method is used, there will always be a distribution shift to the model input induced by the deformation T-1, A well chosen noise process will simply try to minimize this shift as much a possible.
The work [4] proposes to use diffusion models trained on discrete inputs distributed according to N(0, Ix) and computes conditional distributions < (Ex)) ) E (E&) using the stochastic interpola- tion method described above, generalized to two dimensions. We, instead, propose to use Gaussian process (0, Q). as described Section and and compute e < (Ex) (E))))(())))) by conditioning this process which amounts to simply evaluating the RFF projection. It is our numerical experience that this better preservers the qualitative properties of the input distribution for large deformations, We leave the exploration of process best suited for this task as important future work.
D Additional Results
In this section, we provide additional results that did not fit in the main paper. We visualize the difference between independent noise and noise from GP in Figure We present inpainting results from our SDXL. inpainting model fine-tuned with GP noise in Figure We present super- resolution results from our SDXL super-resolution model fine-tuned with GP noise in Figure [ We further present warping errors with respect to the previous frame in Figure for the inpainting results and warping errors for super-resolution for real videos in Figure Finally, we present additional comparisons for super-resolution in Figure TO
E
Related Works
Our work is primarily related to three recent lines of research about the utility of diffusion models in inverse problems, video editing, and equivariance in function space diffusion models as elaborated below.
Diffusion Models for Inverse Problems. Diffusion models have been recently received widespread adoption for solving inverse problems in various domains. Diffusion models can solve inverse problems in a few different ways. A simple way is to train or finetune a conditional diffusion model for each specific task learn the conditional distribution from the degraded data distribution the clean data distribution [7] 53, (7). Some popular examples include SR3 (75) and inpainting stable diffusion [67]. We leverage stable diffusion inpainting the present work. While successful, they however need to be trained (or finetuned) separately for each individual task that is computationally complex. Also, they are not robust to out of distribution data. To mitigate these challenges, plug- and-play methods have been introduced that utilize single foundation diffusion model (e.g., stable diffusion) as a (rich) prior to solve many inverse problems once this approach is to modify the sampling post-hoe by either: (i) add guidance to the score function
70 15] 47]. The crux of
18

---

Inpainting
Inpainting
0.175
Self warping Iatent errar wrt DIay Frame
(gp)
(gp)
(indep)
(nidep)
Ressmple (go)
0.150
Resample Igp)
0.5
Recemple (Indep)
Recemmpl (irdeg)
6P Notsz Warping
9.125
GP Notzo Warping
0.075
Saif warping
0.2
0.000
0.025
30
12
30
12
Frame Indix
Frame Indiex
(a) Warping emor w.r.t. previously generated frame in latent space.
(b) Warping error w.r.t previously generated frame in
pixel space.
Figure 4: Warping emross w.r.t. previously generated frame in latent and pixel space for the inpainting task as we shift the input frame.
Super
Super
Salf warphng lafent ertor Net flrst (ramn
Roed (Gp)
1.75
Resamplt ple (Gp)
0.08
1.25
0.06
1.00
0.50
Fiaed (Cil
0.02
Resample (Gp)
Seif
(Gp)
0.00
Index
Index
(a) Warping error w.r.t. first generated frame latent space.
(b) Warping error w.r.t. first generated frame pixel space.
Self warging latent entor wrt grov prov frame
(Fil
Resample (Gp)
No wo preds (Qp)
0.05
Ours (Gp)
Fixed tGp) (Op)
0.03
Self
30
12
12
Frame Indixx
Frame Indixx
(c) Warping w.r.t. previously generated frame in latent space.
(d) Warping error w.r.t. previously generated frame in pixel space.
Figure 5: Warping errors w.r.t. first generated frame (top-row) and prev. generated frame (bottom row) for the 8x super-resolution task for real videos.
19

---

(a) Independent noise realization.
(b) Gaussian Process noise realization.
Figure 6: Visualization of independent noise and noise from Gaussian Process.
of diffusion models as in 15 (ii) approximated projection onto the measurement subspace at each diffusion step [16 47 or, (ini) use regularization by denoising via optimization [56] [2]. In this work we adopt the guidance-based approach impose equivariance for the score function. All these methods have been applied for 2D images. For video inverse problems, the problem is more challenging due temporal consistency. There are some efforts leverage diffusion models for example for text-to-video superresolution systematic framework yet based on 2D diffusion models to solve generic video inverse problems in temporally consistent manner. This is essentially the focus of our work. Finally, we remark that recent work (22 21 23
inpainting; see e.g., [69 31 193. However, there is no
has shown that it even possible to train diffusion models to solve inverse problems without ever seeing clean images from the distribution of interest.
Video Editing with Image Diffusion Models. Due to the lack of full-fledeen call-fledged pre-train pre-trained text-to-video diffusion models, many works focus on video editing (or video-to-vides translation) using text-to- image diffusson models. One line of research has proposed to fine-tune the image diffusion model on single text-video pair and generate novel videos that represent the edits at inference [88] 54 92
Specifically, Tune-A-Ve A-Video [88 proposed cross-frame attention mechanism and an efficient one- shot tuning strategy. Video-P2P [54] further improved the video inversion performance by optimizing a shared unconditional embedding for all frames. EI2 [92] refined the temporal modules to resolve semantic disparity and temporal inconsistency of video editing. However, the fine-tuning process over the input video makes the editing less efficient. Another line of research has developed various training-fire free methods for efficient video editing, which mostly rely the fross-frame attention and latent fusion for maintaining temporal consistency [13] 51 61 90). In particular, VideoVideo- Zero [5I] encoded the motion dynamics in latent noises through noise wrapping. FateZero [6]] fused the attention features with a blending mask obtained by the source prompt's cross-attention map. VideO [13] proposed to progressively propagate the changes to the future frames via self-attention feature injection. Rerender-A A- Video [90] proposed hierarchical cross- frame constraints with the optical flow for improved temporal consistency.
49 50 have extended diffusion models to function data. However, these methods primarily focus on theoretical developments and have been examined on simplistic datasets such as time series, Navier-Stole solutions, or hand-written digits. This paper can be considered one of the first successful applications of function-space diffusion models to natural image datasets. Our work is also related to the equivariant diffusion models which have been extensively explored in scientific applications such
Function Space Diffusion Models and Equivariance Recently, several works 52
as molecule and protein interaction and generation applications 43
3
17 45). However,
equivariant diffusion models for image generation are less explored, primarily because guaranteeing equivariance (for example with respect to translation, rotation, or rescaling) in commonly used
diffusion architectures such as U-Net [68 39 Transformer [58 36 models is challenging.
Diffusion models trained with correlated noise. Ours is not the first work to train diffusion models with prior other than white noise. The authors of 20 42) show how to train diffusion models
20

---

FINAL VICTORY
ODK VOLAE 1.280VEH
FINAL VICTORY
OOK VOL 1.288YER
ER PRSIIF-AARR RRBC
WHA
21

---

Figure 7: Inpainting examples, Left column: inputs by randomly masking images from the COYO dataset. Right column: inpainting outputs from our SDXL fine-tuned model with correlated noise.
with blurring corruption, leading blurred terminal distribution. Several other works have shown how to generalize diffusion models to find mappings between arbitrary input-output distributions, including 66] [0O] 24]. One new finding our work is that is possible start with a state-of-the-ant model trained with white noise and fine-tune it casily to handle correlated noise. This allows us to
convert vanilla diffusion models to Function Space Diffusion models by training them with noise sampled from Gaussian Processes. For more details, we refer the reader to Section 3.1
F
Experimental Details
F11 Dealing with Optical Flows
We use the RAFT model to predict the optical flows 83 respect to the first frame between subsequent frames. We find that the optical flow estimation is much better between subsequent frames and we use subsequent transformations to find the position in the original frame, whenever possible.
The optical flows can be computed with
Since we are working with Latent Diffusion Models, all the warping happens lower-dimensional space. Fortunately, as observed in numerous prior works, including [57), there is geometric
22

---

23

---

Salary
Salary
30000.0
30000.0
40000.0
40000.0
50000.0
50000.0
60000 n
60000 0
53
53
M
MY
R
R
DY
DY
Figure 8: Super-resolution examples. Left column: downsampled inputs from the COYO dataset. Right column: super-resolution outputs from our SDXL fine-tuned model with correlated noise.
correspondence between pixel blocks and latent locations. i.c. pixel blocks are mapped to specific locations in latent space. This allows us to extract the flows from the input frames and convert them to optical flows our latent vectors. Alternatively, one nat first map to latent space and then compute the optical flow there. We did not pursue this approach since we rely on deep learning method for the fow-estimation and the underlying model has been trained on natural images.
F.2 Stable Diff'usion XL Finetuning
To fine-tune SDXL in conditional tasks, we use the reference implementation found in the following link: hww.t/hh..oo/nngggggc/////iiiiis
The reference implementation finetunes SDXL on the inpainting task, however, is straightforward to adapt other conditional tasks, such as super-resolution. As mentioned in the paper, we train all our models for 100, 000 steps. We use following training hyperparameters:
Training resolution: 1024 x 1024.
Batch size: 64.
Latent resolution: 128.
24

---

ODE sampling trajectories
frame j
Equivariance Self-Guidance
oingoon
frame j+1
Figure 9: Schematic visualization of Equivariance Self Guidance (see Algorithm
Super-resoution Shifts
Super-rssolution Over
Set warping plxe) emor WTT praw Romm
Fleed Noise Indep
Resample Nekie indep
0.010
0.010
How Marped Your Noise
holse
0.008
0.008
0.008
0.006
0.004
Food tride
0.004
Resample Netse Indep
How Warrped Yuur Noise GP Noise Warping
0.000
Frame Index
(a) Warping w.r.t. previously generated frame in pixel space.
Warping w.r.t. first generated frame in pixel space.
Figure 10: Warping errors in pixel space for the super-resolution task as we shift the input frame.
Optimizer Adam with Weight Decay. Optimizer parameters:
Learning rate: 5e
S1 = 0.9
32 0.999
Weight Decay: le
le- 08
Max Gradient Norm (Gradient clipping): 1.0
Gaussian Process parameters:
1. Truncation parameter: 2.0
2. Number of random features: 3000
3. Length scale: 0.004977.
The parameter length scale controls the amount correlation the noise from the GP. Recall that RFFs are generated by sampling 2; ~ ((, c (0)) To avoid aliasing artifacts when generating GP. we truncated the Normal distribution at 2 -1 (i.e., 2x its standard deviation) and we made sure that
-1 is lower than the Nyguist-Shannen sampling frequency, i.c., 2e resolucion
Given this, as
2
25

---

general rule of thumb, we found that setting the length scale to be 6 x-resolution leads to noise realizations that can used to easily fine-tune Stable Diffusion XL.
We train all our models on 16 A100 GPUs on a SLURM-hased cluster. The fine-tuning of the SDXL model on conditional tasks (super-resolution, inpainting) with correlated noise for 100k steps takes roughly 24 hours.
F.3 Sampling Speed
Sampling guidance for equivariance increases the generation time for two reasons: i) need to more steps in order to make effective and, ii) each step is more expensive since we need to perform additional hackpropagation. For experiments, for unconditional sampling. Further, without guidance, we get 4.32 iterations per second on a single A100 GPU while with guidance obtain 1.62 iterations per second.
50 steps instead of 25 steps that
The other hyperparameter used sampling the guidance strength, see Algorithm | For À there is no guidance and the method just becomes GP Noise Warping. For higher À the gradient from warping guidance becomes stronger. In our experiments, we found the value À = to perform the best. This is consistent with the choice of À in the Diffusion Posterior Sampling [15 paper which uses guidance term to apply diffusion models for general inverse problems.
We perform all our sampling experiments takes roughly 20 seconds to generate single frame. We measure the performance our method and the baselines on second videos consisting of 16 frames.
single A-100 GPU. Without sampling guidance, it
G Broader Impact
Our method allows the use of image diffusion models to solve video inverse problems. There are both positive and negative societal implications of such method. On the positive side, our method does not require training of video models which is typically expensive and contributes to increasing the Al carbon footprint. Further, democratizes access to video editing tools. The average practitioner can now leverage state-of-the-an image models to solve video inverse problems. To illustrate the effectiveness of our method, we trained powerful text-conditioned inpainting models that work arbitrary images from the web. On the negative side, these models can be used for adversarial image and video editing. Further, method can be used for the generation of deepfakes.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The goal of the paper to introduce principled method apply image diffusion models to solve temporally correlated inverse problems. This is clearly stated in both the abstract and the introduction. The method developed the rest of the paper and is supported by experimental evidence.
Justification:
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We list the Limitations of our work in Section S
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We clearly state Theoretical statements and proofs in Section A in the Appendix.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We do our best to make our framework reproducible possible. We list all the details regarding the fine-tuning of Stable Diffusion XI. in the Experiments Section and in Section F in the Appendix. We analytically describe how sample correlated noise in Section 3.1 Finally, we provide a reference implementation for our Equivariance Self-Guidance in A Algorithm
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: For our fine-tuning experiments, we simply adapt the code found in the following link: https://giiuub.oo//ugggeg http://ugnngacoccdffusssspp////p
We do not yet release our code for noise warping and sampling guidance, but we are working on open-sourcing
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the most essential parts of our experimental setup in Section of the paper and we give further details, including the choice of our hyperparameters, in Section F2 of the Appendix.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
27

---

Justification: We include error bars for all our Figure plots (see Figures 3 and we also report the standard deviation for all the evaluation metrics our tables (see Tables 2
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: Yes, see Section F.2
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:/:/oorrrpssccc//pllaaaaaas
Answer: [Yes]
Justification: We have carefully reviewed and conformed with the NeurIPS Code of Ethics. We have made sure that our submission preserves anonymity.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of our work in Section G in the Appendix.
11. Safeguards
Question:
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The main asset we use a pre-trained Stable Diffusion model. explicitly
mention its version (Stable Diffusion XL) and we cite the corresponding paper.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release any assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
28


Reviews:
[{"review_id": "IPjffRLBRs", "reviewer": "Reviewer_zjYi", "summary": "This paper proposes a self-guidance equivariance approach using the Image Diffusion model for generating temporally consistent videos. In previous work HOW I WARPED YOUR NOISE[1], noise is warped across frames to ensure the noise maps are temporally consistent. However, temporally consistent noise maps do not guarantee temporally consistent output. In this work, the authors address this issue. The proposed self-guidance equivariance enforces that the generated frames remain consistent under the warping transformation. The proposed method can generate more consistent results. In their experiments, the authors conducted comparisons on video inpainting and superresolution. It is worth noting that the proposed method outputs consistent content in the inpainting area across frames.", "questions": "- A better way to illustrate how the proposed Equivariance Self-Guidance works.\n  - A proper figure would be better instead of just formulations.\n\n- See Point 2 in Weakness; can the proposed method handle cases where the object is moving or has different actions in the video inpainting case instead of being still?", "limitations": "For the limitations, the authors mention that in some situations, the work may fail. It would be helpful to also provide the corresponding failing examples, such as the second and third examples mentioned in Appendix F Limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Problem Setup\n  - The problem seems reasonable and important, especially in the case of video inpainting. Compared to other video applications, any temporary inconsistent case can create a dramatic failure case. See Figure 1. \n\n- Temporal consistent results. \n    - In this work, the authors show how the proposed equivariance self-guidance improves video inpainting results and superresolution.\n\n- More efficient noise warping.\n   - Compared to the previous work [How I Warped Your Noise], this work improves the inference time by up to 16 times.", "weaknesses": "- Better illustration and explanation of how the `self-guidance Equivariance` works.\n  The main novelty of the paper is the self-guidance equivariance, which improves performance compared to previous work, HOW I WARPED YOUR NOISE [1]. However, it is not easy to understand how it works in Figure 2. The current Figure 2 only shows its role and the input and output in the overall system but does not describe its function. Besides, this figure seems to provide more information about how to use optical flow to guide the warping process of noise.\n\n- Inpainting examples and usefulness?\n  Do the warped noise and self-guidance Equivariance work for generating \"moving objects\" in video inpainting? According to the formulation and the generation results, it seems it will favor generating \"still objects\" in the video inpainting case because it will easily meet the \"warped\" mechanism.\n\n- Experiment Design.\n  Experiments in Section 4.1 are not needed (or do not provide a useful argument) in the paper. Previous works have shown that warped noise is better than independent noise across the frames in a video. The only contribution here is that the fine-tuned version performs better than the un-fine-tuned version. The finding seems unrelated to the paper or provides quite limited information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their thoughtful review. We are glad to see that the Reviewer appreciated many aspects of our submission, such as the importance of the problem we are addressing and the experimental benefits we are obtaining. In what follows, we do our best to address any remaining concerns. \n\n`[...] it is not easy to understand how it works in Figure 2. [...] A proper figure would be better instead of just formulations.`\n\nWe thank the Reviewer for their comment. We agree that Figure 2 should be improved. Following the recommendation of the Reviewer, we updated the figure to show that the self-guidance is applied in each step of the ODE sampler. The figure is now available at the (anonymous) project website and at [this link](https://bashify.io/i/veUk9P). We plan to include this update in the camera-ready version of our work. If there is any more feedback, we would be glad to incorporate it.\n\n`Do the warped noise and self-guidance Equivariance work for generating \"moving objects\" in video inpainting? According to the formulation and the generation results, it seems it will favor generating \"still objects\" in the video inpainting case because it will easily meet the \"warped\" mechanism.`\n\nWe thank the Reviewer for their insightful question.  The proposed framework should work for moving objects in video inpainting as long as the optical flow map is available and there are no occlusions. That said, as the Reviewer correctly points out, for realistic videos, the estimation of the optical flow map might be noisy. We will make this point more explicit in our paper and we will include videos of moving objects in the camera-ready version of our work. We thank again the Reviewer for bringing up this important point.\n\n`Experiments in Section 4.1 are not needed (or do not provide a useful argument) in the paper. Previous works have shown that warped noise is better than independent noise across the frames in a video. The only contribution here is that the fine-tuned version performs better than the un-fine-tuned version. The finding seems unrelated to the paper or provides quite limited information.`\n\nWe respectfully disagree with the Reviewer on this point. Typically, diffusion models are trained with i.i.d. noise. In Section 4.1, we show that one can finetune a state-of-the-art diffusion model to work with correlated noise as input. Before finetuning, Stable Diffusion XL will produce unrealistic images when the sampling chain is initialized with correlated noise. Our experiments show that post-finetuning, the model can handle spatially correlated noise in the input without compromising performance. To the best of our knowledge, spatially correlated noise has not been shown effective in prior work (note that this is different from prior works on video generation that use temporally correlated noise). Our GP Warping mechanism requires models that can handle correlated noise. Hence, these fine-tunings are essential for the rest of the paper. We will clarify this point in the main paper to avoid confusing the reader. \n\n`For the limitations, the authors mention that in some situations, the work may fail. It would be helpful to also provide the corresponding failing examples, such as the second and third examples mentioned in Appendix F Limitations.`\n\nWe definitely agree with the Reviewer and we thank them for encouraging us to make negative results publicly available.\n\n* Equivariance of the decoder. Sometimes, the decoder is not equivariant with respect to the underlying transformation. We noticed that this is a common failure for text rendering. This can be better illustrated with an example. The generated latent video [here](https://anonneurips2024.github.io/assets/videos/3_output_latent_video.mp4) appears equivariant. However, the decoded version of this video, found [here](https://anonneurips2024.github.io/assets/videos/3_output_video.mp4) is not equivariant. Specifically, the produced text changes from one frame to the other.\n\n* Correlation artifacts: For extreme deformations, there is a distribution shift between training and inference which leads to correlation artifacts. This has already been observed in the prior work, How I Warped Your Noise. In [this](https://warpyournoise.github.io/docs/assets/videos/SuperRes/Bear/adv.mp4) example from the project website, there appears to be a colored wave near the back of the bear. This is an example of a correlation artifact. Such artifacts also appear sometimes in our work, e.g. see the texture artifacts that appear in the final frame of [this](https://anonneurips2024.github.io/assets/videos/216_output_video.mp4) generated video.\n\nWe will definitely include these failure cases in our camera-ready version and we thank again the Reviewer for raising this important point. **We hope our response addresses your concerns and that you consider upgrading your rating.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oL8hQNeirQ", "reviewer": "Reviewer_WsM3", "summary": "The paper follows a previous work on noise-warping methods for video inverse problem. The authors proposed instead of giving temporally consistent noise maps to the DM, using a continuous function space representation for DM directly can serve more complex spatial transformations and results in better temporal consistency in the generated videos. They demonstrated the proposed method on video inpainting and super-resolution.", "questions": "1. It is not clear to me why the function space diffusion models need to be equivariant with respect to the underlying spatial transformations. I can imagine the necessity, but in your derivation, I don't see the proof that you stated you provide. Can you provide explicit examples or solid proof on this statement?\n2. Why in fig 3 how i warped your noise seems to be in the second place in terms of error, but in table 2, the method seems not to outperform any other baselines?", "limitations": "The authors did not address any limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed method is novel in terms of using Gaussian processes to handle the noise function so that no mapping is required for the noise. The authors derived the equivariance for deformation and by satisfying this the gaussian process is used to guide the generation to be consistent under the warping transformation. To avoid additional training, a sampling mechanism was proposed to ensure this satisfaction at inference time.", "weaknesses": "The key statement in the paper is the importance of the equivariance of the DM to achieve temporal consistancy. The authors argued that the previous work \"How I Warped Your Noise\" performed badly when the DM is not equivariant. However, there are no explicit examples or solid proof on this statement. The proposed method for noise warping is more like offering another perspective than showing the failing reasons of the other work.\n\nFurthermore, for super-resolution experiments, the authors did not provide comparison against \"How I warped your noise\". Even though they elaborated that the authors of \"How I warped your noise\" acknowledged to them that the work does not result in temporal consistency, I believe it is still better to include the comparison given how many super-resolution cases were presented in that work. If they don't provide the code, the authors may consider modify the one they have for the inpainting experiments and list out the comparison for more convincing results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their time and their valuable review. We are glad that the Reviewer appreciated the novelty of our work. In what follows, we do our best to answer some remaining questions.\n\n`The authors argued that the previous work \"How I Warped Your Noise\" performed badly when the DM is not equivariant. However, there are no explicit examples or solid proof on this statement. The proposed method for noise warping is more like offering another perspective than showing the failing reasons of the other work.`\n\nOur framework has two components: (1) the GP warping scheme, and (2) the self-guidance method at inference time. In Figure 3 of the paper, we show that (1) is not enough: both GP noise warping and the How I Warped Your Noise baseline still have significant warping error. Self-guidance is essential to further decrease the warping error because the DM is not equivariant. This is also illustrated in Row 2 of Figure 1: the How I Warped Your Noise warping scheme fails to produce consistent results because the underlying DM is not equivariant. If the Reviewer wants further clarification on this point, we would be happy to provide it.\n\n`Furthermore, for super-resolution experiments, the authors did not provide comparison against \"How I warped your noise\". Even though they elaborated that the authors of \"How I warped your noise\" acknowledged to them that the work does not result in temporal consistency, I believe it is still better to include the comparison given how many super-resolution cases were presented in that work. If they don't provide the code, the authors may consider modify the one they have for the inpainting experiments and list out the comparison for more convincing results.`\n\nWe thank the Reviewer for their suggestion! We provide comparisons for the super-resolution task with How I Warped Your Noise in the case of integer translation in the following anonymous links: \n1) [(Self-) Warping Error with respect to previous generated frame in pixel space](https://anonneurips2024.github.io/assets/self_warping_pixel_error_wrt_prev_frame.pdf)\n2) [(Self-) Warping Error with respect to first frame in pixel space](https://anonneurips2024.github.io/assets/self_warping_pixel_error_wrt_first_frame.pdf)\n\nThese results are similar to what's given in Figure 3 of the paper and will be included in the camera-ready version of our paper. Once again, we thank the Reviewer for suggesting this important experiment.\n\n`Why in fig 3 how i warped your noise seems to be in the second place in terms of error, but in table 2, the method seems not to outperform any other baselines?`\n\nWe believe the Reviewer might have misunderstood this part. Figure 3 shows the Warping Error and the How I Warped Your Noise paper gets the second (best) place. This is consistent with the first column (Warping Error) of Table 2 which places again the GP in the second best place in terms of warping error. The rest of the columns of Table 2 measure the quality of the generated images (but not temporal consistency across frames). \nAny warping mechanism sacrifices some quality for temporal consistency. This has been also reported in the How I Warped Your Noise paper (see Table 1, page 7 of the paper). Our results in Table 2 are consistent with this finding. Hopefully, this clarifies the question raised by the Reviewer.\n\n`The authors did not address any limitations`\n\nWe would like to bring to the Reviewer’s attention Section F of our Appendix. In this Section, we state several limitations of our work. To increase the visibility of this Section, we will bring it to the main paper in the next-revision of our work.\n\n`It is not clear to me why the function space diffusion models need to be equivariant with respect to the underlying spatial transformations.`\n\nThe formalization of this statement follows from the definition of the video that we use in this paper. We define a video such that the next frame is given as a deformation of the last frame. If the generative model is not equivariant w.r.t. this transformation, then its output will no longer satisfy this definition of a video. We argue that since optical flows are a useful tool for video modeling and have been empirically successful, this definition of a video, through the optical flow, is a reasonable one. In terms of empirical proof, Figure 1 shows that the robot keeps changing if you don't have equivariance.\n\n**We hope our response addresses your concerns and that you consider increasing your rating.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6GT27gznnZ", "reviewer": "Reviewer_kW4a", "summary": "This paper addresses the challenge of temporally correlated inverse problems by employing an image diffusion model. The authors introduce a technique termed Warped Diffusion, which incorporates an equivariance self-guidance mechanism. This mechanism ensures that the generated frames maintain consistency when subjected to warping transformations. The paper demonstrates the application of this technique in the context of video inpainting and video super-resolution through a series of experiments.", "questions": "See the weaknesses.", "limitations": "The authors have adequately discussed the limitations and negative societal impact in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is presented in a structured manner, with illustrations that aid in understanding. The content is coherent and follows a logical sequence.\n\n2. The authors offer an analysis of noise warping and equivariance self-guidance, providing both intuitive explanations and theoretical underpinnings.\n\n3. 3. The experiments are extensive to support the claim.", "weaknesses": "1. The authors report the noise warping speed at Line 357. We recommend that the authors also include the complete inference time required to process an entire video, which would provide a more comprehensive understanding of the method's efficiency.\n\n2. We suggest that the authors provide more video results to further demonstrate the capabilities and limitations of the proposed method in various scenarios.\n\n3. It is observed that the inpainted regions in the video inpainting examples remain static, such as the cat in the video not exhibiting motion. All apparent motion seems to be attributed to camera movements, with no optical flow present within the inpainted regions. We would appreciate an explanation for this phenomenon and its implications on the video inpainting process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their time and insightful review. We are very glad that the Reviewer appreciated the presentation of our work, the theoretical underpinnings of our formulation and the strong empirical performance of our method.\n\n`The authors report the noise warping speed at Line 357. We recommend that the authors also include the complete inference time required to process an entire video, which would provide a more comprehensive understanding of the method's efficiency.`\n\nWe have some additional discussion regarding sampling time in Section G.3. of the Appendix. Processing a 2-second video takes roughly ~5 minutes on a single A-100 GPU. We will move this discussion to the main paper, as recommended by the Reviewer.\n\n`We suggest that the authors provide more video results to further demonstrate the capabilities and limitations of the proposed method in various scenarios.`\n\nWe thank the Reviewer for their suggestion. We include more results on the [project webpage](https://anonneurips2024.github.io/) as recommended by the Reviewer. We will de-anonymize this webpage and make it available to the public upon acceptance of our work.\n\n`It is observed that the inpainted regions in the video inpainting examples remain static, such as the cat in the video not exhibiting motion. All apparent motion seems to be attributed to camera movements, with no optical flow present within the inpainted regions. We would appreciate an explanation for this phenomenon and its implications on the video inpainting process.`\n\nWe thank the Reviewer for raising this question. Our proposed framework is not constrained to static examples and can work with real optical flows, as demonstrated for the super-resolution task. Inpainting is a much harder inverse problem that super-resolution and thus maintaining consistency for real flows might be more challenging. We did not experiment with real video inpainting in this paper but we plan to include some examples for the camera-ready version.\n\nWe hope our response addresses your concerns and that you consider increasing your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nt4jNb90hQ", "reviewer": "Reviewer_ebKQ", "summary": "This paper aims to address video inverse problems using image diffusion models. By viewing videos as sequences of warping transformations between frames, the authors propose a novel approach called warped diffusion from a continuous function space perspective. Specifically, warped diffusion includes a Gaussian process-based noise-warping technique and a post-hoc equivariance self-guidance method to enhance temporal consistency. Experiments on video super-resolution and inpainting validate the effectiveness of warped diffusion. Moreover, warped diffusion first demonstrates superior performance with latent diffusion models such as SDXL.", "questions": "1. \"How I Warped Your Noise\" operates in a zero-shot manner using pretrained video editing diffusion models without requiring additional training. Is it possible for warped diffusion to also function in a zero-shot manner?\n2. I am curious about the effectiveness of warped diffusion with just the equivariance self-guidance alone, without the GP noise warping.\n3. What would the performance be like if the equivariance self-guidance technique were integrated into \"How I Warped Your Noise\"? Would it surpass the current performance of warped diffusion?", "limitations": "The limitations and societal impacts have already been discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well-motivated with well-constructed formulations.\n2. The paper is well-written.\n3. The visualization results are impressive.", "weaknesses": "1. As illustrated in Table 1 and Figure 3, the proposed GP Noise Warping performs worse than \"How I Warped Your Noise\" in terms of warping error. The core performance improvement appears to come from the equivariance self-guidance.\n2. As mentioned in Section G.3, the equivariance self-guidance is inefficient. Although the proposed GP Noise Warping is more efficient than \"How I Warped Your Noise,\" the overall process can be more time-consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their time and their high-quality review. We are glad that the Reviewer appreciated the importance of the problem we are solving, the presentation of our work, and our experimental results. In what follows, we do our best to answer the questions raised by the Reviewer.\n\n`As illustrated in Table 1 and Figure 3, the proposed GP Noise Warping performs worse than \"How I Warped Your Noise\" in terms of warping error. The core performance improvement appears to come from the equivariance self-guidance.`\n\nWe believe there is some misunderstanding here. In Table 1, no noise warping is happening (hence no comparison with the How I Warped Your Noise work). The reported results are for the generation of the first frame of a video. The results show that finetuning with correlated noise does not compromise performance. We will clarify this further to avoid confusing the reader. \n\nIn Figure 3, we measure equivariance with respect to translation by an integer amount of pixels. For this deformation, the How I Warped Your Noise warping mechanism and the proposed GP Warping are essentially the same: both mechanisms just shift the input noise by the appropriate number of pixels. This is explained in Lines 315 to 318 in the paper. We intentionally chose to compare on this task because it simplifies the How I Warped Your Noise method's implementation (for which no reference code is available).\n\nSince for this task, GP Noise Warping and How I Warped Your Noise are essentially the same algorithm, the reason there is some small performance difference is that we are applying the warping to different models: the GP Noise Warping uses the finetuned model with correlated noise while the How I Warped Your Noise paper uses the finetuned model with independent noise. We will clarify this point in the paper and we thank a lot the Reviewer for raising it.\n\n`As mentioned in Section G.3, the equivariance self-guidance is inefficient. Although the proposed GP Noise Warping is more efficient than \"How I Warped Your Noise,\" the overall process can be more time-consuming.`\n\nWe agree with the Reviewer that the overall process can be more time-consuming if guidance is used. Our paper has two major contributions: (1) a principled framework for noise warping (GP Warping) and (2) a guidance method to enforce the equivariance of the underlying network (self-guidance). The How I Warped Your Noise paper is a warping method and hence when we compare the speed with this baseline, we measure the time needed for warping (see Lines 357). Our warping method (contribution (1)) is 16x faster and it generates images at higher resolution. That said, it is true that sampling guidance (contribution (2)) will significantly increase the sampling time. Namely, as we mention in Section G.3., we need ~2.67x the time needed without guidance for every single step and we run it for twice the number of steps, leading to an overall ~5.34x slowdown compared to not using guidance. For full transparency, we will move this discussion to the main paper and we thank the Reviewer for bringing up this important point.\n\n`\"How I Warped Your Noise\" operates in a zero-shot manner using pretrained video editing diffusion models without requiring additional training. Is it possible for warped diffusion to also function in a zero-shot manner?`\n\nThis is a very important point and we thank the Reviewer for raising it. Our method cannot work in a zero-shot manner since it requires a model that is trained with correlated noise. We mention this in the paper, but we will explicitly add it to the Limitations Section (Section F). As we show in the paper, with minimal finetuning it is possible to get a model trained with independent noise to work with correlated noise (Table 1), but still, some amount of finetuning is required, as the Reviewer correctly pointed out.\n\n`I am curious about the effectiveness of warped diffusion with just the equivariance self-guidance alone, without the GP noise warping.`\n\nThis is a great question. We ran some preliminary experiments for super-resolution on real-videos and we found that omitting the warping significantly deteriorates the results when the number of sampling steps is low. We found that increasing the number of sampling steps makes the effect of the initial noise warping less significant, at the cost of increased sampling time. We will include this discussion in the camera-ready version of our paper and we thank a lot the Reviewer for raising this important point.\n\n`What would the performance be like if the equivariance self-guidance technique were integrated into \"How I Warped Your Noise\"? Would it surpass the current performance of warped diffusion?`\n\nThis is an excellent question. We believe that adding self-guidance to the How I Warped Your Noise paper would significantly improve the results. Unfortunately, there is no way to check this for general transformations because there is no reference implementation for us to try. We tried this for the integer shift transformation (for which the implementation becomes trivial) and the How I Warped Your Noise + self-guidance method matched the performance of our proposed algorithm. We will include this experiment in the camera-ready version of our work and we thank a lot the Reviewer for raising this insightful question.\n\nWe hope our response addresses your concerns and that you consider increasing your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All the 4 reviewers recommend acceptance. The authors should add discussions on failure cases, improve the organization and the presentation of the manuscript, and include the additional results and clarifications into the final copy.
Decision: Accept (poster)