Title: Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: noise warping

inverse problems

diffusion models

equivariance
Abstract: Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos as a sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only on **images** and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the underlying spatial transformations. To ensure temporal consistency, we introduce a simple post-hoc test-time guidance towards (self)-equivariant solutions. Our method allows us to deploy state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and $8\times$ video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results in the following URL: https://giannisdaras.github.io/warped_diffusion.github.io/.

Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models

Abstract

Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos as a sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only on images and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the underlying spatial transformations. To ensure temporal consistency, we introduce a simple post-hoc test-time guidance towards (self)-equivariant solutions. Our method allows us to deploy state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and 8 × video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results in the following URL: https://giannisdaras.github.io/warped_diffusion.github.io/.

1 Introduction

Diffusion models (DMs) [79, 39, 82] can synthesize photorealistic imagery [73, 64, 66, 5, 60, 26]. They can be conditioned easily, through explicit training or guidance [25, 41], and have also been widely used to solve inverse problems [19, 86, 15, 16, 80, 84, 47, 56], in particular for image processing applications like inpainting and super-resolution [40, 72, 74, 66].

How do these methods extend to video processing and solving inverse problems on videos? Although video DMs are seeing rapid progress [38, 78, 9, 29, 8, 30, 7, 11], general textto-video synthesis has not yet reached the level of robustness and expressivity comparable to modern image models. Moreover, no state-of-the-art video generative models are publicly available [11], and

How I Warp

(Ours)

most video DMs are computationally expensive. To circumvent these challenges, a natural research direction is to leverage existing, powerful image generative models to solve video inverse problems.

Naively applying image DMs to videos in a frame-wise manner violates temporal consistency. Previous works alleviate the problem by fine-tuning on video data or by warping the networks' features, using, for instance, temporal or cross-frame attention layers [88, 54, 13, 51, 61, 90, 92, 34, 33]. However, these methods are usually designed specifically for high-level text-driven editing or stylization and are typically not directly applicable to general inverse problems. Moreover, without training on diverse video data they often cannot maintain high frequency information across frames. For a detailed discussion of the related works, we refer the reader to Section E in the Appendix.

The recent novel work, 'How I Warped Your Noise' [14], proposes noise warping to achieve temporal consistency in generated videos by changing appropriately the input noise to the diffusion model. Videos can be thought of as image frames subject to spatial transformations. An object may move according to a translation; complex and general transformations can be described by motion vectors on the pixels defined through optical flow [27]. It is these transformations that define how the noise maps need to be warped and transformed. In [14], temporally consistent noise maps are given as input to the DM's denoiser, with the underlying assumption that temporally consistent inputs induce temporally consistent network outputs. In this paper, we argue that this assumption only holds true if the utilized image DM is equivariant with respect to the spatial warping transformations. However, as we show in this work, the network is not necessarily equivariant because i) the conditional expectation modeled by the DM may not be equivariant, and, ii) more importantly, a free-form neural network, as used in typical DMs, will not learn a perfectly equivariant function. When the equivariance assumption is violated, the method proposed in [14] achieves poor results. This is typically the case for challenging conditional tasks (see Figure 1) or when modeling complex distributions. Particularly, [14] finds that the proposed method has 'limited impact on temporal coherency' when applied to latent diffusion models and that 'all the noise schemes produce temporally inconsistent results'.

We introduce a new framework, dubbed Warped Diffusion , for the rigorous application of image DMs to video inverse problems. We employ a continuous function space perspective to DMs [52, 59, 28, 35] that naturally allows noise warping for arbitrarily complex spatial transformations. Our method generalizes the warping scheme of [14] and does not require any auxiliary high-resolution noise maps. To achieve equivariance, we propose equivariance self-guidance , a novel sampling mechanism that enforces that the generated frames are consistent under the warping transformation. Our inference time approach elegantly circumvents the need for additional training. This unlocks the use of existing large DMsin a fully equivariant manner without further training, which may be prohibitive for a practitioner.

We extensively validate our method on video inpainting and super-resolution. Super-resolution represents a situation with strong conditioning, while inpainting requires large-scale, temporally coherent synthesis of new content. Warped Diffusion outperforms previous methods quantitatively and qualitatively, and shows reduced flickering and texture sticking artifacts. Due to our equivariance guidance, our method can also be used with latent DMs, which is not possible with previous approaches. Virtually all existing state-of-the-art text-to-image generation systems are indeed latent DMs, like Stable Diffusion [66]. Hence, any inverse problem solving method must be readily usable with latent DMs. In fact, all our experiments utilize the state-of-the-art text-to-image latent DM SDXL [60].

Contributions: (a) We propose Warped Diffusion, a novel framework for applying image DMs to video inverse problems. (b) We introduce a principled scheme for noise warping, based on Gaussian processes and a function space DM perspective. (c) We identify the equivariance of the DM as a critical requirement for the seamless application of image DMs to video inverse problems and propose an inference-time guidance method to enforce it. (d) We comprehensively test Warped Diffusion and achieve state-of-the-art video processing performance when considering the use of image DMs. Critically, Warped Diffusion can be used with any image DMs, including large-scale latent DMs.

2 Functional Video Generation

The basis of our approach, summarized in Figure 2, is to structure the generative model so that it is equivariant with respect to spatial deformations and apply these deformations successively to the input noise. Each deformation effectively warps the noise and the equivariance guarantees that each output image will be similarly warped. By using an optical flow from a real video to define a sequence of such deformations, a new video can be generated. To introduce our method, we first conceptualize both images and noise as functions on a domain and the generator as a mapping between two function spaces.

Low-Resolution

Optical

(b)

Equivariance Self-Guidance

Input Frames

Flow

ODE sampling steps

Warping

GPNoise

Previous

Samples

Frame

OpticalFlow

Error per Step

ODESampler

Self-Guidance

Equivariance

d

ODESampler

Current

Frame

gami ninja

High-Resolution

Output Frames

2.1 Functional Generative Modeling and Videos

Each video frame can be seen as a single image, and an image as a discretization of a vector-valued function on a rectangular domain. Consider the domain as the 2 -D unit square D = [0 , 1] 2 , defining an image as a function f : D → R 3 . For each location x ∈ D , the value f ( x ) ∈ R 3 represents an RGB color. We assume images have infinite resolution. To formulate a model that generates such images, we must have a notion of a space containing all possible images. We'll use the separable Hilbert space H = L 2 ( D ; R 3 ) , with pointwise formulas interpreted almost everywhere with respect to the Lebesgue measure.

We assume that there exists a probability measure µ on H whose support is the set of photorealistic images and denote by η a known reference probability measure on H . In our case, η will be a Gaussian measure on H ; for details, see Section 3.1. A generative model , or transport map , is then a mapping G : H → H such that the pushforward of η under G is µ which we denote as G ♯ η = µ . In particular, this implies that any random variable ξ ∼ η will satisfy G ( ξ ) ∼ µ . For diffusion models, G can be defined by the probability flow ODE; see Section 3.2.

Given an image f 0 ∈ H , a video with n +1 ∈ N frames is the sequence of functions ( f 0 , f 1 , . . . , f n ) ∈ H n +1 , where each subsequent function is obtained, at least partially, from the previous one by a deformation. Specifically, a sequence of bounded, injective maps ( T j : D → D j ) n j =1 exists such that

2.2 Video Generation and Equivariance

Given our notion of a video and a generative model, we now describe how such a model can be used to generate new videos. Suppose we want to create a two-frame video given an initial frame f 0 ∈ H and a deformation map T 1 : D → D 1 . Assume we have a generative model G : H → H and an initial noise image ξ 0 ∈ H such that G ( ξ 0 ) = f 0 . From definition, the new frame of our video is

f 1 = f 0 ◦ T -1 1 on D ∩ D 1 . If D ⊆ D 1 , it might seem that our generative model is unnecessary. However, proceeding this way generates blurry and unrealistic videos.

The primary issue is that, in practice, we don't have access to f 0 at an infinite resolution but only at a fixed, finite set of grid points E k = { x 1 , . . . , x k } ⊂ D . To determine f 1 on our grid points, we need the values of f 0 at the points T -1 1 ( E k ) = { T -1 1 ( x 1 ) , . . . , T -1 1 ( x k ) } . It's highly unlikely that E k = T -1 1 ( E k ) for any realistic deformation.

Thus, we must interpolate f 0 to T -1 1 ( E k ) , which usually leads to blurry results with standard methods. Furthermore, if D ̸⊆ D 1 , there will be regions where f 1 is not determined by f 0 and will need to be inpainted on the new visible domain. Therefore, for each frame, we must solve an interpolation and an inpainting problem: tasks for which generative models are well-suited.

Suppose we have access to the noise function ξ 0 at infinite resolution, and its domain extends to all of R 2 ; we discuss both in Section 3.1. We can then define the new frame in our video by applying the generative model to the deformed noise: f 1 = G ( ξ 0 ◦ T -1 1 ) . The deformed noise function ξ 0 ◦ T -1 1 gets its values from ξ 0 | D for points in D ∩ D 1 and from the extension of ξ 0 to R 2 for all other points where inpainting is needed. To ensure this definition is consistent with (1), G must be equivariant with respect to T -1 1 . Specifically, for all ξ ∈ supp ( η ) ⊆ H , we must have

Assuming (2), it follows from G ( ξ 0 ) = f 0 , that f 1 ( x ) = G ( ξ 1 | D )( x ) = ( f 0 ◦ T -1 1 )( x ) for all x ∈ D ∩ D 1 hence the pair ( f 0 , f 1 ) is a valid 2 frame video according to the definition of Section 2.1. To generate a video with any number of frames, we simply iterate on this process with a given sequence of deformation maps. Enforcing (2) can be done directly by the architectural design, through training with various deformation maps, or, through a guidance process; see Section 3.2.

2.3 White Noise

It is common practice to train generative models assuming the reference measure η is Gaussian white noise. Specifically, a draw ξ ∼ η on the grid points E k = { x 1 , . . . , x k } ⊂ D is realized as ξ ( x l ) = χ l for an i.i.d. sequence χ l ∼ N (0 , 1) for l = 1 , . . . , k . However, this approach is incompatible with our goal of having the generative model perform interpolation. For most deformations T encountered in practice, none of the points in T -1 ( E k ) will match those in E k . Consequently, each new evaluation ξ ( T -1 ( x l ) ) will be independent of the sequence { χ l } k l =1 , making ξ ( T -1 ( E k ) ) appear as a new noise realization unrelated to ξ ( E k ) . This incompatibility arises because white noise processes are distributions, not regular functions, meaning realizations are almost surely not members of H [18]. [14] proposes a stochastic interpolation method to address this issue (see Appendix C for details and comparison). We generalize this idea and propose using generic Gaussian processes on H .

3 Method: Warped Diffusion

In Section 1, we formulated the problem of video generation as the computation of a series of functions warped by an optical flow and proposed the use of a generative model for inpainting and interpolating the warped functions. The main challenges which remain are defining a functional noise process which can be evaluated continuously and a generative model which is equivariant with respect to warping. We propose to use Gaussian processes for our functional noise and a guidance procedure within the sampling step of a diffusion model to overcome these challenges.

3.1 Gaussian Processes (GPs)

A Gaussian Process (GP) η is a probability measure on H completely specified by its mean element and covariance operator. For a mathematical introduction, see Appendix B. We identify Gaussian processes with positive-definite kernel functions κ : R 2 × R 2 → R . Recall that E k = { x 1 , . . . , x k } denotes the grid points where we know the values of an image f ∈ H . To realize a random function ξ ∼ η on these points, we sample the finite-dimensional multivariate Gaussian N (0 , Q ) , where Q ∈ R k × k is the kernel matrix Q ij = κ ( x i , x j ) for i, j = 1 , . . . , k .

Once sampled, given the fixed values ξ ( E k ) , ξ can be evaluated at any new point x ∗ ∈ D by computing the conditional distribution ξ ( x ∗ ) | ξ ( E k ) [65]. This approach allows us to realize

random functional samples at infinite resolution through conditioning, thus resolving the interpolation problem. Furthermore, by ensuring the kernel κ is positive definite on a domain larger than D , we can consistently sample ξ outside of D , addressing the inpainting problem described in Section 2.2.

For high-resolution images when k is large, working with the matrix Q can be computationally expensive. Instead, we propose using R andom F ourier F eatures (RFF) to sample η , which amounts to a finite-dimensional projection of the function ξ ∼ η that converges in the limit of infinite features [63, 87]. We can approximate samples from a GP with a squared exponential kernel with length-scale parameter ϵ > 0 by ξ ( x ) = √ 2 J ∑ J j =1 w j cos ( ⟨ z j , x ⟩ + b j ) for i.i.d. sequences w j ∼ N (0 , 1) , z j ∼ N (0 , ϵ -2 I 2 ) , b j ∼ U (0 , 2 π ) where J ∈ N is the number of features. RFF allows us access to ξ at infinite resolution on the entirety of the plane while also allowing for efficient computation.

3.2 Function Space Diffusion Models and Equivariance Self-Guidance

We will now focus on the generative model that needs to be equivariant to the noise transformations. Specifically, in this section, i) we introduce function space diffusion models, ii) we prove that if every prediction of the diffusion model is equivariant then the whole diffusion model sampling chain is equivariant to the underlying spatial transformations, and, iii) we describe equivariance self-guidance , our sampling technique for enforcing the equivariance assumption.

For ease of notation, we will present everything for the case of unconditional video generation. However, our method seamlessly incorporates any addition conditioning information that may be available. If c 0 , . . . , c n ∈ R c is a sequence of known conditioning vectors then these can simply be passed into a conditional score model at the appropriate frame without any other change to our method; see Algorithm 1. Conditioning vectors could be, for example, low resolutions versions of a video or an original video with regions masked. In Section 4, we focus on such conditional tasks.

Function Space Diffusion Models. Typically, diffusion models are trained with white noise. As explained in Section 2.3, a principled continuous evaluation of the noise requires a functional process. We briefly describe diffusion models in the context of sampling using the Gaussian processes of Section 3.1. We show in Section 4.1 (Table 1) that a model trained with white noise can be fine-tuned to GP noise without any loss in performance.

While it is possible to formulate diffusion models on the infinite-dimensional space H e.g. [52], we will proceed in the finite-dimensional case for ease of exposition. In particular, we will define the forward and backward process as a flow on a vector u ∈ R k , thinking of the entries as the values of a scalar function evaluated on the grid E k and recall that Q is the kernel matrix on E k .

We consider forward processes of the form,

where W t is a standard Wiener process on R k and σ is a scalar-valued, once differentiable function. This process results in conditional distributions p ( u t | u 0 ) = N ( u 0 , σ 2 ( t ) Q ) , see [46]. Let p ( u t , t ) denote the density of u t induced by (3). Then the following backward in time ODE,

started at u ( τ ) distributed according to (3) has the same marginal distributions p ( u t , t ) as (3) on the interval [0 , τ ] ; see [46]. Approximating N ( u 0 , σ 2 ( τ ) Q ) by N (0 , σ 2 ( τ ) Q ) , we may then define the generative model G by the mapping u ( τ ) ↦→ u (0) with reference measure η = N (0 , σ 2 ( τ ) Q ) .

Solving (4) requires knowledge of the score ∇ u log p ( u t , t ) . Instead of learning the score, we opt for directly learning the weighted score Q ∇ u log p ( u t , t ) . This design choice leads to faster sampling since we do not need to perform any expensive matrix multiplication with Q at inference time.

A generalized version of Tweedie's formula (for proof see Appendix A.2) implies:

We approximate E [ u 0 | u t ] with a neural network h θ by minimizing the denoising objective:

Having a minimizer h θ of (6) gives us access to the weighted score Q ∇ u log p ( u t , t ) via (5). We may then obtain an approximate solution to the map u ( τ ) ↦→ u (0) by discretizing (4) in time. We consider Euler scheme updates given by

started with u τ ∼ N (0 , σ 2 ( τ ) Q ) for some time step ∆ t > 0 .

Equivariance for the Probability Flow ODE. Since the diffusion model works with discrete inputs, we need to introduce a discretization of (2) for the network. For a deformation T 1 , we define equivariance as

which is obtained from composing both sides of (2) with T 1 . Note that (8) is valid only for pixels which stay within frame and we compute the l.h.s. with bilinear interpolation on the network output. The input to the network on the l.h.s. is computed with RFFs without any interpolation. Given this discrete equivariance is satisfied for every prediction of the network, it is straightforward to show that the whole diffusion model sampling chain will be equivariant. Indeed, the whole approximation to u ( τ ) ↦→ u (0) is equivariant by the linearity of composition - for a full derivation, see Appendix A.3.

Equivariance Self-Guidance. The condition (8) is rarely satisfied for deformations T 1 arising in practical settings. This is because either the conditional expectation E [ u 0 | u t ] is not equivariant with respect to T -1 1 or the neural network approximation has not fully captured it. If the underlying equivariance assumption breaks, methods that rely solely on noise warping for temporal consistency, e.g. [14], will perform poorly. This is evident in challenging conditional tasks (see Figure 1).

A potential solution is to directly train the network by adding (8) as a regularizer. However, this requires large amounts of video data from which to extract optical flows. Furthermore, by satisfying (8) over a large class of T 1 (s), the network may become less apt at satisfying (6) and lose its generative abilities. Therefore, we opt for guiding the model towards equivariant solutions at inference time .

We first sample noise u (0) τ and generate the first frame following (7), keeping the outputs of the network at each time step { h θ ( u (0) t , t ) } . To generative the next frame, we warp our noise u (1) τ = u (0) τ ◦ T -1 1 with RFFs and again follow (7) but this time using (8) as guidance. In particular, we take a gradient steps in the direction of the loss function | h θ ( u (1) t , t ) ◦ T 1 -h θ ( u (0) t , t ) | 2 , computed on the pixels that stay within frame. All frames can be generated by iterating this procedure as summarized in Algorithm 1 (and visualized in Figures 2, 9) which also shows how to use conditioning information. Guidance is typically used to solve inverse problems with diffusion models (e.g. see [15]), but here the guidance is applied to align the model with its own past predictions. We emphasize that to compute the composition with T 1 above, we use bilinear interpolation on the network outputs but we never need to interpolate the network inputs since we can compute the warping via RFFs. Furthermore, since we are matching interpolated outputs to ones that are not interpolated, our output images remain sharp. This in contrast to directly using a discrete version of (2) which would suggest that we match network outputs to interpolated images, producing blurry results.

4 Experimental Results

For all our experiments, we use Stable Diffusion XL [60] (SDXL) as our base image diffusion model. We start by finetuning SDXL on conditional tasks. We choose super-resolution and inpainting as the tasks of interest since they are both commonly used in the inverse problems literature and they represent two distinct scenarios: in super-resolution, the input condition is strong and in inpainting, the model needs to generate new content. For super-resolution, we choose a downsampling factor of 8 . For inpainting, we create masks of different shapes at random, following the work of [57]. During the finetuning, we train the model to predict the uncorrupted image given the following inputs: i) the encoding of the noised image, ii) the noise level, and, iii) the encoding of the corrupted (downsampled/masked) image. To condition on the corrupted observation, we concatenate the measurements across the channel dimension. We train models with and without correlated noise on the COYO dataset [12] for 100 k steps. We show realizations of independent and correlated noise in Figure 6. Additional implementation details are in Section F.2, including the parameters for the GP introduced in Section 3.1.

4.1 Training with correlated noise

The first step is to assess the quality of the trained models. To do so, we take images from a test split of the COYO dataset, we corrupt them (either by masking or downsampling) and we measure the conditional performance of the trained models. We use a diverse set of metrics that are commonly used in the inverse problems literature: CLIP Text Score [62], CLIP Image Score [62], SSIM [85], LPIPS [91], MSE, Inception Score [76] and FID [37]. The first five metrics measure point-wise restoration performance. Inception Score measures the quality of the generated distribution (without an explicit reference distribution). Finally, FID measures restoration performance in a distributional sense, i.e. it measures how close is the distribution after restoration to the ground truth distribution.

Wereport our results for the super-resolution and inpainting models in Table 1. The main finding is that finetuning with correlated noise does not compromise performance, i.e. SDXL models finetuned with correlated noise perform on par with SDXL models that are trained with independent noise. Particularly for inpainting, the GP models slightly outperform models trained with independent noise across all metrics. We provide qualitative results for our all models in Figure 1 and in Appendix Figures 7, 8.

We remark that the advantages of using an initial distribution other than white noise have been explored in prior work [20, 6, 42]. Our new finding is that a model initially trained with white noise can be easily fine-tuned to work with correlated noise. To the best of our knowledge, ours is the first work that shows that Stable Diffusion XL can be fine-tuned to work with correlated noise.

We underline that prior to fine-tuning Stable Diffusion XL produces unrealistic images when the sampling chain is initialized with correlated noise. Our experiments show that post-finetuning, the model can handle spatially correlated noise in the input without compromising performance. Our GP Warping mechanism requires models that can handle correlated noise. Hence, these fine-tunings are essential for the rest of the paper.

4.2 Noise Warping and Equivariance Self Guidance

In the previous experiments, we measured the restoration performance of the trained models for a single image and we established that models trained with correlated noise perform on par (or even outperform) models trained with independent noise. The next step is to measure the temporal behavior of the models, i.e. how well they work for videos.

Noise Warping baselines. As explained in Section 1, to apply image diffusion models to videos, we need to transform the noise as we move from one frame to the next. We consider the following noise-warping baselines that were used in [14]: Fixed noise uses the same noise across all the frames. Resample noise samples a new noise for each new frame. Nearest Neighbor uses the noise of the nearest location in the grid to evaluate the noise at the location that is not on the regular grid E k . Bilinear Interpolation interpolates the values of the noise bilinearly in the neighboring locations that lie on the grid. How I Warped Your Noise [14] is the state-of-the-art method for solving temporally correlated inverse problems with image diffusion models. It warps the noise by using auxiliary high-resolution noise maps (see our intro, related work section, and Section C). Our GP Noise Warping warps the input noise by resampling the Gaussian process in the mapped locations. We note that the Fixed Noise, Resample Noise, and Nearest Neighbor noise warping methods can be applied to models that are trained with either independent noise or correlated noise coming from a GP. For all the experiments, we also include our proposed method, Warped Diffusion that uses GP Noise Warping and Equivariance Self-Guidance (see Algorithm 1 for a reference implementation).

Video Evaluation Metrics. We follow the evaluation methodology of the 'How I Warped Your Noise' paper [14]. Specifically, we want to measure two different aspects of our method: i) average restoration performance across frames, ii) temporal consistency. For i), we measure the average of all the previously reported metrics (FID, Inception, CLIP Image/Text score, SSIM, LPIPS and MSE) across the frames. For ii), we measure the self-warping error, i.e. how consistent are the model's predictions across time. The warping error can be computed in either pixel or latent space and also with respect to the first generated frame or the previously generated frame, totaling 4 warping errors.

To warm up, we start with videos that are synthetically generated by 2-D shifting of a single image, as in Figure 1. To further simplify the setup, we consider the easy case of shifting the current frame by an integer amount of pixels with each new frame. For 2-D translations by an integer amount of pixels, the Nearest Neighbor, Bilinear Interpolation, How I Warped Your Noise and GP Noise Warping methods they become essentially the same since we always evaluate the noise distribution on points in the grid E k . Hence, the only difference is whether we apply these methods to white noise or to GPs.

Figure 1 (Row 2) shows that the How I Warped Your Noise baseline produces temporally inconsistent results as we shift the masked input image. Even though all the inpaintings are of high quality, the baseline results are temporally inconsistent. Instead, our Warped Diffusion method produces temporally consistent results since it enforces equivariance by design. Since the How I Warped Your Noise warping mechanism and GP coincide here, the benefit strictly comes from enforcing the equivariance property. In fact, one could get the same results for the How I Warped Your Noise method by penalizing for equivariance at inference time.

We present quantitative results regarding temporal consistency in Figure 3 (and additional results in Figure 4 in the Appendix). As shown in the Figure, the fixed noise and the resample noise baselines perform the worst w.r.t. the temporal consistency both in latent and pixel space. The warping error of the Resample baseline is almost constant across frames as expected, while the warping error of the Fixed Noise increases with time. Both the How I Warped Your Noise method and our GP warping framework significantly improve the baselines. Yet, they still have significant temporal inconsistencies as evidenced by the results in Figure 1 and the supplemental videos. The two methods perform on par on this task since they are essentially the same when it comes to integer shifts: the only difference is that GP Noise Warping is applied to correlated noise coming from a GP. The remaining temporal errors are not an artifact of the noise warping mechanism but they are due to the fact that the model itself is not equivariant w.r.t. the underlying transformation. The warping errors essentially disappear when we apply Equivariance Self Guidance. As shown in Figure 3, our method, Warped Diffusion, achieves almost 0 warping error (1e-4 mean pixel error with respect to the first frame to be precise) since it is enforcing equivariance by design.

The only remaining question is whether Warped Diffusion maintains good restoration performance. To answer this, we measure mean restoration performance across frames for the aforementioned metrics. We report our results in Table 2, including the mean warping error with respect to the first frame. As shown, Warped Diffusion maintains high performance across all the considered metrics while being significantly superior in terms of temporal consistency. The conclusion is that all the other noise warping baselines, including the previous state-of-the-art How I Warped Your Noise paper [14], perform poorly in terms of temporal consistency since they rely on the assumption that the network is equivariant. Even for simple temporal correlations such as integer movement in the 2-D space, this

Self warping latent error wrt first frame

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

0

2

4

Inpainting

Fixed (gp)

Fixed (indep)

Resample (gp)

Resample (indep)

Ours (gp)

How i warped your noise (indep)

6

8

10

12

14

Frame Index

(a) Self-warping error w.r.t. first frame in latent space.

0

2

4

6

8

10

12

14

Frame Index

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Self warping pixel error wrt first frame

Inpainting

Fixed (gp)

Fixed (indep)

Resample (gp)

Resample (indep)

Ours (gp)

How i warped your noise (indep)

(b) Self-warping error w.r.t. first frame in pixel space.

assumption is false for the challenging inpainting task. Warped Diffusion is the only method that achieves temporal consistency while it still manages to maintain high reconstruction performance.

We finally remark that our sampling algorithm enforces equivariance in the latent space. Yet, the warping errors are negligible in the pixel space as well. Our finding is that improving latent space equivariance translates to improvements in pixel space equivariance. The authors of [14] also find that 'the V AE decoder is translationally equivariant in a discrete way'.

4.3 Effect of Sampling Guidance for more general transformations

We proceed to evaluate our method on realistic videos. We measure performance on 600 captioned videos from the FETV [55] dataset. Since baseline inpainting methods fail even for very simple temporal transformations, we focus on 8 × super-resolution for our comparisons on FETV.

For our video results, we could not provide comparisons with the How I Warped Your Noise paper. At the time of this writing, there was no available reference implementation as we confirmed with the authors by direct communication. In any case, the authors acknowledge as a limitation of their work that their proposed method has 'limited impact on temporal coherency' when applied to latent models and that 'all the noise schemes produce temporally inconsistent results' [14]. Once again, we attribute this to the non-equivariance of the denoiser, which we mitigate with our guidance algorithm.

We proceed to evaluate our method and the baselines with respect to temporal consistency and mean restoration performance across frames, as we did for our inpainting experiments. We present our results in Table 3 and additional results in Figures 5, 8 of the Appendix and in the following URL as videos: https://giannisdaras.github.io/warped_diffusion.github.io/. As shown in Table 3, there is a trade-off between temporal consistency and restoration performance. Methods that perform better in terms of temporal consistency often have significantly worse performance across the other metrics. Our Warped Diffusion achieves a sweet spot: it has the lowest warping error by a large margin and it still maintains competitive performance across all the other metrics. On the contrary, methods that are based solely on noise warping, such as GP Warping and the simple interpolation methods, lead to significant performance deterioration for a small improvement in temporal consistency.

Noise Warping Speed. We measure the time needed for a single noise warping. Our GP Warping mechanism takes 39 ms per frame Wall Clock time, to produce the warping at 1024 × 1024 resolution. This is 16 × faster than the reported 629 ms number in [14]. If we use batch parallelization, our method generates 1000 noise warpings in just 46 ms (at the expense of extra memory).

No Warping? A natural question is whether we can omit completely the noise warping scheme since equivariance is forced at inference time. We ran some preliminary experiments for super-

resolution on real-videos and we found that omitting the warping significantly deteriorates the results when the number of sampling steps is low. We found that increasing the number of sampling steps makes the effect of the initial noise warping less significant, at the cost of increased sampling time.

5 Limitations

Our method has several limitations. First, the guidance term increases the sampling time, as detailed in the Appendix, Section F.3. For reference, processing a 2-second video takes roughly 5 minutes on a single A-100 GPU. Second, even though in our experiments we observed a monotonic relation between the warping error in latent space and warping error in pixel space, it is possible that for some transformations the decoder of a Latent Diffusion Model might not be equivariant. We noticed that this is a common failure for text rendering, e.g. in this latent video the model seems to be equivariant, but in the pixel video it is not. Third, the success of our method depends on the quality of the flow estimation - inconsistent flow estimation between frames will lead to flickering artifacts. For real videos, there might be occlusions and the estimation of the flow map can be noisy. We observed that in such cases our method fails, especially for challenging tasks such as video inpainting. The correlations obtained by following the optical flow field obtained from real videos might lead to a distribution shift compared to the training distribution. For such extreme deformations, our method produces correlation artifacts. This has been observed in prior work (see this video), but it also appears in our setting (e.g. see this video). Finally, our method cannot work in a zero-shot manner since it requires a model that is trained with correlated noise.

6 Conclusions

Warped Diffusion is a novel framework for solving temporally correlated inverse problems with image diffusion models. It leverages a noise warping scheme based on Gaussian processes to propagate noise maps and it ensures equivariant generation through an efficient equivariance self-guidance technique. We extensively validated Warped Diffusion on temporally coherent inpainting and superresolution, where our approach outperforms relevant baselines both quantitatively and qualitatively. Importantly, in contrast to previous work [14], our method can be applied seamlessly also to latent diffusion models, including state-of-the-art text-to-image models like SDXL [60].

References

Asad Aali, Marius Arvinte, Sidharth Kumar, and Jonathan I Tamir. Solving inverse problems with score-based generative priors learned from noisy data. arXiv preprint arXiv:2305.01166 , 2023.

Asad Aali, Giannis Daras, Brett Levac, Sidharth Kumar, Alexandros G Dimakis, and Jonathan I Tamir. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data. arXiv preprint arXiv:2403.08728 , 2024.

Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019 , 2022.

Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society , 68, 1950.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392 , 2022.

Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 , 2024.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In Advances in Neural Information Processing Systems , 2021.

Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.

Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset , 2022.

Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23206-23217, 2023.

Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C Azevedo. How i warped your noise: a temporally-correlated noise prior for diffusion models. In The Twelfth International Conference on Learning Representations , 2023.

Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 , 2022.

Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00941 , 2022.

Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776 , 2022.

G. Da Prato and J. Zabczyk. Stochastic Equations in Infinite Dimensions . Encyclopedia of Mathematics and its Applications. Cambridge University Press, 2014.

Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Peyman Milanfar, Alexandros G. Dimakis, Chul Ye, and Mauricio Delbracio. A survey on diffusion models for inverse problems. 2024.

Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05442 , 2022.

Giannis Daras and Alex Dimakis. Solving inverse problems with ambient diffusion. In NeurIPS 2023 Workshop on Deep Learning and Inverse Problems , 2023.

Giannis Daras, Alexandros G Dimakis, and Constantinos Daskalakis. Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data. arXiv preprint arXiv:2404.10177 , 2024.

Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems , 36, 2024.

Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303.11435 , 2023.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems , 2021.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

Denis Fortun, Patrick Bouthemy, and Charles Kervrann. Optical flow modeling and computation: A survey. Computer Vision and Image Understanding , 134:1-21, 2015.

Giulio Franzese, Giulio Corallo, Simone Rossi, Markus Heinonen, Maurizio Filippone, and Pietro Michiardi. Continuous-time functional diffusion processes. Advances in Neural Information Processing Systems , 36, 2024.

Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709 , 2023.

Ramesh Girish and Gopal Krishna. A survey on video diffusion models. arXiv preprint arXiv:2310.10647 , 2023.

Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-andplay priors. arXiv preprint arXiv:2206.09012 , 2022.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933 , 2023.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations (ICLR) , 2024.

Paul Hagemann, Sophie Mildenberger, Lars Ruthotto, Gabriele Steidl, and Nicole Tianjiao Yang. Multilevel diffusion: Infinite dimensional score-based diffusion models for image generation. arXiv preprint arXiv:2303.04772 , 2023.

Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. arXiv preprint arXiv:2312.02139 , 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282 , 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arXiv:2209.05557 , 2022.

Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning (ICML) , 2022.

Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems , 34:14938-14954, 2021.

Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. Advances in Neural Information Processing Systems , 35:24240-24253, 2022.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems , 35:23593-23606, 2022.

Bahjat Kawar, Noam Elata, Tomer Michaeli, and Michael Elad. Gsure-based diffusion model training with corrupted data. arXiv preprint arXiv:2305.13128 , 2023.

Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions. arXiv preprint arXiv:2212.00886 , 2022.

Gavin Kerrigan, Giosue Migliorini, and Padhraic Smyth. Functional flow matching. arXiv preprint arXiv:2305.17209 , 2023.

Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15954-15964, 2023.

Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.07400 , 2023.

Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar. I 2 sb: Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.05872 , 2023.

Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with crossattention control. arXiv preprint arXiv:2303.04761 , 2023.

Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. arXiv preprint arXiv: 2311.01813 , 2023.

Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations , 2023.

Suraj Patil. Sdxl inpainting model, 2024. Accessed: 2024-05-21.

William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748 , 2022.

Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimensional diffusion models. arXiv preprint arXiv:2302.10130 , 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations (ICLR) , 2024.

Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15932-15942, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning . Adaptive Computation and Machine Learning series. MIT Press, 2005.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752 , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI) , 2015.

Claudio Rota, Marco Buzzelli, and Joost van de Weijer. Enhancing perceptual quality in video super-resolution through temporally-consistent detail synthesis using diffusion models. arXiv preprint arXiv:2311.15908 , 2023.

Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2111.05826 , 2021.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636 , 2021.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence , 45(4):4713-4726, 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Conditional simulation using diffusion schrödinger bridges. arXiv preprint arXiv:2202.13460 , 2022.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Video: Text-toVideo Generation without Text-Video Data. In The Eleventh International Conference on Learning Representations (ICLR) , 2023.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , 2015.

Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations , 2022.

Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations , 2023.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 402-419. Springer, 2020.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 , 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint arXiv:2112.02475 , 2021.

James Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Deisenroth. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning , pages 10292-10302. PMLR, 2020.

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7623-7633, 2023.

Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations (ICLR) , 2022.

Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers , pages 1-11, 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Zicheng Zhang, Bonan Li, Xuecheng Nie, Congying Han, Tiande Guo, and Luoqi Liu. Towards consistent video editing with text-to-image diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Shangchen Zhou et al. Flow-guided diffusion for video inpainting. arXiv preprint arXiv:2311.13752 , 2023.

A Theoretical Results

A.1 Convolutions and Equivariance

To better understand (2), we consider the following example. We will assume that ξ : R 2 → R is a scalar-valued field (grayscale image) defined on the whole plane. Furthermore, we let G be given by a continuous convolution and T -1 1 be a translation. In particular, for all x ∈ R 2 ,

for some compactly supported kernel κ : R 2 → R and a direction a ∈ R 2 . We then have

by the change of variables formula. This shows that G is equivariant to all translations. This is a well-known property of the convolution and, in particular, it shows that convolutional neural networks

are translation equivariant, noting that pointwise non-linearities will preserve this property. This example shows that a model can be equivariant with respect to certain deformations by architectural design. However, in realstic video modeling, optical flows are not know explicitly and can only be approximated numerically. It is therefore natural to instead build-in approximate equivariance into a model instead of enforcing it directly in the architecture. Our guidance procedure in Section 3.2 is an example such an approximate form of equivariance.

A.2 Tweedie's Formula

In Section 3.2, we show a diffusion model can be trained and sampled from using Gaussian process noise instead of white noise. Our result depends on the following lemma which is a simple generalization of Tweedie's formula.

Lemma A.1. Let x be a random variable with positive density p x ∈ C 1 ( R k ) . Let σ > 0 and z ∼ N (0 , Q ) for some positive definite matrix Q ∈ R k × k and assume that x ⊥ z . Define the random variable

and let p y ∈ C ∞ ( R k ) be the density of y . It holds that

Proof. First note that by the chain rule,

where p ( y, x ) denotes the joint density of ( y, x ) . Let p ( y | x ) denote the Gaussian density of the conditional y | x . Since p x ∈ C 1 ( R k ) ,

Therefore, by the chain rule,

Since p ( y | x ) is the density of N ( x, σ 2 Q ) , a direct calculations shows that

Furthermore, Bayes' theorem implies

as desired.

A.3 Flow Equivariance

In Section 3.2, we claim that if the score network h θ is equivarient with respect to a deformation T -1 , then the Euler scheme approximation of the map u ( τ ) ↦→ u (0) is equivarient with respect to T -1 . It is easy to see that this results holds so long as it holds for the single step u t ↦→ u t -∆ t defined by (7). We will assume that h θ safisfies (8) written as

We make sense of this equation by using RFF to define u t as a function on the plane and similarly bilinear interpolation to define h θ ( u t , t ) as a function. It follows by linearity of composition that

which is the requisite equivariance of the map u t ↦→ u t -∆ t .

Therefore,

B Gaussian Processes

A probability measure η on H is called Gaussian if there exists an element m ∈ H and a self-adjoint, non-negative, trace-class operator Q : H → H such that, for all h, h ′ ∈ H ,

where ⟨· , ·⟩ denotes the inner product on H . The element m is called the mean while the operator Q is called the covariance . It is immediate from this definition that white noise is not included since the identity operator is not trace-class on any infinite dimensional space. This definition ensures that any realization of a random variable ξ ∼ η is almost surely an element of H . When the domain D of the elements of H is a subset of the real line, η is often called a Gaussian process . We continue to use this terminology even when D is a subset of a higher dimensional space i.e. R 2 but remark that the nomenclature Gaussian random field is sometimes preferred.

Since we working on a separable space, each such field on H has associated to it a unique reproducing kernel Hilbert space [18, Theorem 2.9] which is associated to a unique positive definite kernel [4]. In particular, there exists a positive definite function κ : D × D → R for which Q is its associated integral operator. It follows that a Gaussian process can be uniquely identified with a positive definite kernel. Sampling and conditioning this process can then be accomplished via the kernel matrix.

To make this explicit, suppose that X = { x 1 , . . . , x n } ⊂ D and Y = { y 1 , . . . , y m } ⊂ D are two sets of points in D . We will slightly abuse notation and write

for the kernel matrix between X and Y and similarly Q ( Y, X ) , Q ( X,X ) , Q ( Y, Y ) . Suppose that ξ ∼ η is a random variable from the Gaussian process with kernel κ and mean zero. To sample a realization of ξ on the points X , we sample the finite dimensional Gaussian N ( 0 , Q ( X,X ) ) . This can be written as

where Z ∼ N (0 , I n ) . Suppose now that the points in Y are distinct from those in X and we want to sample ξ on Y given the realization ξ ( X ) . This can be done by conditioning [65]

While the above formulas fully characterize sampling ξ , working with them can be computationally burdensome. It is therefore of interest to consider a different viewpoint on Gaussian processes, in particular, through the Karhunen-Loève expansion. The spectral theorem implies that Q possesses a full set of eigenfunctions Q j ϕ j = λ j ϕ j for j = 1 , 2 , . . . with some decaying sequence of eigenvalues λ 1 ≥ λ 2 ≥ . . . . The random variable ξ ∼ N (0 , Q ) can be written as

where χ j ∼ N (0 , 1) is an i.i.d. sequence and the right hand side sum converges almost surely in the norm of H [18]. By truncating this sum to a finite number of terms, computing realizations of ξ becomes much more computationally manageable. This inspires the random features approach to Gaussian processes which is the basis of our computational method; for precise details, see [65, 63, 87].

C Brownian Bridge Interpolation

We show in Section 2.3 that a white noise process is not compatible with the idea of using a generative model to interpolate deformed functions. A potential way of dealing with this issue is to treat the original realizations of the white noise ξ ( E k ) as the fixed nodal points of a function ξ and obtain the rest of the values via interpolation. It is shown in [14] that common forms of interpolation yield a conditional distribution ξ ( T -1 ( E k ) ) | ξ ( E k ) that is too dissimilar from the training distribution N (0 , I k ) and thus the generative model produces blurry or disfigured images.

Therefore [14] proposes a stochastic interpolation method which has the property that, for a new point x ∗ ̸∈ E k , the distribution of ξ ( x ∗ ) marginalized over the joint distribution ( ξ ( E k ) , ξ ( x ∗ ) ) follows N (0 , 1) . This is most easily seen in one spatial dimension with k = 2 points. Suppose that D = [0 , 1] and let a, b ∼ N (0 , 1) be two independent random variables. Consider a Gaussian process on D with kernel function κ ( x, y ) = 1 -| x -y | and suppose that ξ is distributed according to this GP conditioned on ξ (0) = a and ξ (1) = b . A straightforward calculation shows that, for any x ∗ ∈ (0 , 1) ,

for z ∼ N (0 , 1) independent of ( a, b ) . This is simply the Brownian bridge connecting a to b . Remarkably, the marginal distribution of ξ ( x ∗ ) over the joint ( ξ ( x ∗ ) , a, b ) is N (0 , 1) independently of x ∗ . However, the conditional distribution is

which is not N (0 , 1) for all x ∗ ∈ (0 , 1) . In [14, Section 2.2], it is proposed that such Brownian bridges are used between any two pair of pixels, yielding a stochastic interpolation method given by a sequence of such independent GPs. However, from the point of view of using a generative model that is pre-trained on N (0 , I k ) , it is not of interest that the marginal distribution of ξ ( x ∗ ) is N (0 , 1) but rather that the conditional ξ ( x ∗ ) | a, b is N (0 , 1) . As we have seen, this is not the case for the method of [14] and, in fact, it will only ever be the case for white noise processes as discussed in Section 2.3. Therefore, no matter what method is used, there will always be a distribution shift to the model input induced by the deformation T -1 . A well chosen noise process will simply try to minimize this shift as much a possible.

The work [14] proposes to use diffusion models trained on discrete inputs distributed according to N (0 , I k ) and computes conditional distributions ξ ( T -1 ( E k ) ) | ξ ( E k ) using the stochastic interpolation method described above, generalized to two dimensions. We, instead, propose to use a Gaussian process N (0 , Q ) , as described in Section 3.1 and compute ξ ( T -1 ( E k ) ) | ξ ( E k ) by conditioning this process which amounts to simply evaluating the RFF projection. It is our numerical experience that this better preservers the qualitative properties of the input distribution for large deformations. We leave the exploration of a process best suited for this task as important future work.

D Additional Results

In this section, we provide additional results that did not fit in the main paper. We visualize the difference between independent noise and noise from our GP in Figure 6. We present inpainting results from our SDXL inpainting model fine-tuned with GP noise in Figure 7. We present superresolution results from our SDXL super-resolution model fine-tuned with GP noise in Figure 8. We further present warping errors with respect to the previous frame in Figure 4 for the inpainting results and warping errors for super-resolution for real videos in Figure 5. Finally, we present additional comparisons for super-resolution in Figure 10.

E Related Works

Our work is primarily related to three recent lines of research about the utility of diffusion models in inverse problems, video editing, and equivariance in function space diffusion models as elaborated below.

Diffusion Models for Inverse Problems. Diffusion models have been recently received widespread adoption for solving inverse problems in various domains. Diffusion models can solve inverse problems in a few different ways. A simple way is to train or finetune a conditional diffusion model for each specific task to learn the conditional distribution from the degraded data distribution to the clean data distribution [71, 53, 77]. Some popular examples include SR3 [75] and inpainting stable diffusion [67]. We leverage stable diffusion inpainting in the present work. While successful, they however need to be trained (or finetuned) separately for each individual task that is computationally complex. Also, they are not robust to out of distribution data. To mitigate these challenges, plugand-play methods have been introduced that utilize a single foundation diffusion model (e.g., stable diffusion) as a (rich) prior to solve many inverse problems at once [44, 70, 15, 47]. The crux of this approach is to modify the sampling post-hoc by either: ( i ) add guidance to the score function

0

2

4

6

8

10

12

Frame Index

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Self warping latent error wrt prev frame

Inpainting

Fixed (gp)

Fixed (indep)

Resample (gp)

Resample (indep)

Ours (gp)

How i warped your noise (indep)

0

2

4

6

8

10

12

Frame Index

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Self warping pixel error wrt prev frame

Inpainting

Fixed (gp)

Fixed (indep)

Resample (gp)

Resample (indep)

Ours (gp)

How i warped your noise (indep)

0

2

4

6

8

10

12

14

Frame Index

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Self warping latent error wrt first frame

Super res

Fixed (Gp)

Resample (Gp)

No wp preds (Gp)

Ours (Gp)

0

2

4

6

8

10

12

Frame Index

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Self warping latent error wrt prev frame

Super res

Fixed (Gp)

Resample (Gp)

No wp preds (Gp)

Ours (Gp)

0

2

4

6

8

10

12

14

Frame Index

0.00

0.02

0.04

0.06

0.08

0.10

Self warping pixel error wrt first frame

Super res

Fixed (Gp)

Resample (Gp)

No wp preds (Gp)

Ours (Gp)

0

2

4

6

8

10

12

Frame Index

0.01

0.02

0.03

0.04

0.05

0.06

Self warping pixel error wrt prev frame

Super res

Fixed (Gp)

Resample (Gp)

No wp preds (Gp)

Ours (Gp)

(a) Independent noise realization.

of diffusion models as in [15, 81]; ( ii ) approximated projection onto the measurement subspace at each diffusion step [16, 47] or, ( iii ) use regularization by denoising via optimization [56, 32]. In this work we adopt the guidance-based approach to impose equivariance for the score function. All these methods have been applied for 2D images. For video inverse problems, the problem is more challenging due to temporal consistency. There are some efforts to leverage diffusion models for example for text-to-video superresolution or inpainting; see e.g., [69, 31, 93]. However, there is no systematic framework yet based on 2D diffusion models to solve generic video inverse problems in a temporally consistent manner. This is essentially the focus of our work. Finally, we remark that recent work [22, 21, 23, 2, 48, 1] has shown that it is even possible to train diffusion models to solve inverse problems without ever seeing clean images from the distribution of interest.

Video Editing with Image Diffusion Models. Due to the lack of full-fledged pre-trained text-to-video diffusion models, many works focus on video editing (or video-to-video translation) using text-toimage diffusion models. One line of research has proposed to fine-tune the image diffusion model on a single text-video pair and generate novel videos that represent the edits at inference [88, 54, 92]. Specifically, Tune-A-Video [88] proposed a cross-frame attention mechanism and an efficient oneshot tuning strategy. Video-P2P [54] further improved the video inversion performance by optimizing a shared unconditional embedding for all frames. EI 2 [92] refined the temporal modules to resolve semantic disparity and temporal inconsistency of video editing. However, the fine-tuning process over the input video makes the editing less efficient. Another line of research has developed various training-free methods for efficient video editing, which mostly rely on the cross-frame attention and latent fusion for maintaining temporal consistency [13, 51, 61, 90]. In particular, Text2VideoZero [51] encoded the motion dynamics in latent noises through a noise wrapping. FateZero [61] fused the attention features with a blending mask obtained by the source prompt's cross-attention map. Pix2Video [13] proposed to progressively propagate the changes to the future frames via self-attention feature injection. Rerender-A-Video [90] proposed hierarchical cross-frame constraints with the optical flow for improved temporal consistency.

Function Space Diffusion Models and Equivariance Recently, several works [52, 49, 50] have extended diffusion models to function data. However, these methods primarily focus on theoretical developments and have been examined on simplistic datasets such as time series, Navier-Stokes solutions, or hand-written digits. This paper can be considered one of the first successful applications of function-space diffusion models to natural image datasets. Our work is also related to the equivariant diffusion models which have been extensively explored in scientific applications such as molecule and protein interaction and generation applications [43, 3, 89, 17, 45]. However, equivariant diffusion models for image generation are less explored, primarily because guaranteeing equivariance (for example with respect to translation, rotation, or rescaling) in commonly used diffusion architectures such as U-Net [68, 39] or Transformer [58, 36] models is challenging.

Diffusion models trained with correlated noise. Ours is not the first work to train diffusion models with a prior other than white noise. The authors of [20, 42] show how to train diffusion models

La

LW

1.28OYEN

T.286YEN

VICTORY

VICTORY

Voi.12

￥06

F0i.12

with blurring corruption, leading to a blurred terminal distribution. Several other works have shown how to generalize diffusion models to find mappings between arbitrary input-output distributions, including [6, 10, 24]. One new finding in our work is that it is possible to start with a state-of-the-art model trained with white noise and fine-tune it easily to handle correlated noise. This allows us to convert vanilla diffusion models to Function Space Diffusion models by training them with noise sampled from Gaussian Processes. For more details, we refer the reader to Section 3.1.

F Experimental Details

F.1 Dealing with Optical Flows

We use the RAFT model to predict the optical flows [83]. The optical flows can be computed with respect to the first frame or between subsequent frames. We find that the optical flow estimation is much better between subsequent frames and we use subsequent transformations to find the position in the original frame, whenever possible.

Since we are working with Latent Diffusion Models, all the warping happens in a lower-dimensional space. Fortunately, as observed in numerous prior works, including [57], there is a geometric

Salary

Salary

30000.0

30000.0

40000.0

40000.0

50000.0

50000.0

600000

ΛY

ΛY

R

R

DY

DY

correspondence between pixel blocks and latent locations, i.e. pixel blocks are mapped to specific locations in latent space. This allows us to extract the flows from the input frames and convert them to optical flows for our latent vectors. Alternatively, one nat first map to latent space and then compute the optical flow there. We did not pursue this approach since we rely on a deep learning method for the flow-estimation and the underlying model has been trained on natural images.

F.2 Stable Diffusion XL Finetuning

To fine-tune SDXL in conditional tasks, we use the reference implementation found in the following link: https://github.com/huggingface/diffusers/pull/6592. The reference implementation finetunes SDXL on the inpainting task, however, it is straightforward to adapt it to other conditional tasks, such as super-resolution. As mentioned in the paper, we train all our models for 100 , 000 steps. We use the following training hyperparameters:

Training resolution: 1024 × 1024 .

Batch size: 64 .

Latent resolution: 128

.

ODE sampling trajectories

frame j

Equivariance

Self-Guidance

Stellaorigamininjal

frame j+1

0

2

4

6

8

10

12

Frame Index

0.000

0.002

0.004

0.006

0.008

0.010

0.012

Self warping pixel error wrt prev frame

Super-resolution Over Shifts

Resample Noise Indep

How I Warped Your Noise

Ours

(a) Warping error w.r.t. previously generated frame in pixel space.

0

2

4

6

8

10

12

14

Frame Index

0.000

0.002

0.004

0.006

0.008

0.010

0.012

Self warping pixel error wrt first frame

Super-resolution Over Shifts

Resample Noise Indep

How I Warped Your Noise

Ours

(b) Warping error w.r.t. first generated frame in pixel space.

Optimizer Adam with Weight Decay. Optimizer parameters:

Learning rate: 5 e - 6

β 1 = 0 . 9

β 2 = 0 . 999

Weight Decay: 1 e - 2

ϵ = 1 e - 08

-Max Gradient Norm (Gradient clipping): 1 . 0

Gaussian Process parameters:

Truncation parameter: 2 . 0

Number of random features: 3000

Length scale: 0 . 004977 .

The parameter length scale controls the amount of correlation in the noise from the GP. Recall that RFFs are generated by sampling z j ∼ N (0 , ϵ -2 I 2 ) . To avoid aliasing artifacts when generating GP, we truncated the Normal distribution at 2 ϵ -1 (i.e., 2 × its standard deviation) and we made sure that 2 ϵ -1 is lower than the Nyquist-Shannon sampling frequency, i.e., 2 ϵ -1 2 π ≤ resolution 2 . Given this, as

a general rule of thumb, we found that setting the length scale to be ϵ := 2 π · resolution leads to noise realizations that can be used to easily fine-tune Stable Diffusion XL.

We train all our models on 16 A100 GPUs on a SLURM-based cluster. The fine-tuning of the SDXL model on conditional tasks (super-resolution, inpainting) with correlated noise for 100 k steps takes roughly 24 hours.

F.3 Sampling Speed

Sampling guidance for equivariance increases the generation time for two reasons: i) we need to run more steps in order to make it effective and, ii) each step is more expensive since we need to perform an additional backpropagation. For our experiments, we use 50 steps instead of 25 steps that we use for unconditional sampling. Further, without guidance, we get 4 . 32 iterations per second on a single A100 GPU while with guidance we obtain 1 . 62 iterations per second.

The other hyperparameter used in sampling is the guidance strength, see Algorithm 1. For λ = 0 , there is no guidance and the method just becomes GP Noise Warping. For higher λ the gradient from the warping guidance becomes stronger. In our experiments, we found the value λ = 1 to perform the best. This is consistent with the choice of λ in the Diffusion Posterior Sampling [15] paper which uses a guidance term to apply diffusion models for general inverse problems.

We perform all our sampling experiments on a single A-100 GPU. Without sampling guidance, it takes roughly 20 seconds to generate a single frame. We measure the performance of our method and the baselines on 2 second videos consisting of 16 frames.

G Broader Impact

Our method allows the use of image diffusion models to solve video inverse problems. There are both positive and negative societal implications of such a method. On the positive side, our method does not require training of video models which is typically expensive and contributes to increasing the AI carbon footprint. Further, democratizes access to video editing tools. The average practitioner can now leverage state-of-the-art image models to solve video inverse problems. To illustrate the effectiveness of our method, we trained powerful text-conditioned inpainting models that work on arbitrary images from the web. On the negative side, these models can be used for adversarial image and video editing. Further, our method can be used for the generation of deepfakes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The goal of the paper is to introduce a principled method to apply image diffusion models to solve temporally correlated inverse problems. This is clearly stated in both the abstract and the introduction. The method is developed in the rest of the paper and is supported by experimental evidence.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We list the Limitations of our work in Section 5.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We clearly state our Theoretical statements and proofs in Section A in the Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We do our best to make our framework as reproducible as possible. We list all the details regarding the fine-tuning of Stable Diffusion XL in the Experiments Section and in Section F in the Appendix. We analytically describe how we sample correlated noise in Section 3.1. Finally, we provide a reference implementation for our Equivariance Self-Guidance in Algorithm 1.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: For our fine-tuning experiments, we simply adapt the code found in the following link: https://github.com/huggingface/diffusers/pull/6592. We do not yet release our code for noise warping and sampling guidance, but we are working on open-sourcing it.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the most essential parts of our experimental setup in Section 4 of the paper and we give further details, including the choice of our hyperparameters, in Section F.2 of the Appendix.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include error bars for all our Figure plots (see Figures 3, 5) and we also report the standard deviation for all the evaluation metrics in our tables (see Tables 2, 3).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed and conformed with the NeurIPS Code of Ethics. We have made sure that our submission preserves our anonymity.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact of our work in Section G in the Appendix.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The main asset we use is a pre-trained Stable Diffusion model. We explicitly mention its version (Stable Diffusion XL) and we cite the corresponding paper.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address video inverse problems using image diffusion models. By viewing videos as sequences of warping transformations between frames, the authors propose a novel approach called warped diffusion from a continuous function space perspective. Specifically, warped diffusion includes a Gaussian process-based noise-warping technique and a post-hoc equivariance self-guidance method to enhance temporal consistency. Experiments on video super-resolution and inpainting validate the effectiveness of warped diffusion. Moreover, warped diffusion first demonstrates superior performance with latent diffusion models such as SDXL.\n\nSTRENGTHS:\n1. The proposed method is well-motivated with well-constructed formulations.\n2. The paper is well-written.\n3. The visualization results are impressive.\n\nWEAKNESSES:\n1. As illustrated in Table 1 and Figure 3, the proposed GP Noise Warping performs worse than \"How I Warped Your Noise\" in terms of warping error. The core performance improvement appears to come from the equivariance self-guidance.\n2. As mentioned in Section G.3, the equivariance self-guidance is inefficient. Although the proposed GP Noise Warping is more efficient than \"How I Warped Your Noise,\" the overall process can be more time-consuming.\n\nQUESTIONS:\n1. \"How I Warped Your Noise\" operates in a zero-shot manner using pretrained video editing diffusion models without requiring additional training. Is it possible for warped diffusion to also function in a zero-shot manner?\n2. I am curious about the effectiveness of warped diffusion with just the equivariance self-guidance alone, without the GP noise warping.\n3. What would the performance be like if the equivariance self-guidance technique were integrated into \"How I Warped Your Noise\"? Would it surpass the current performance of warped diffusion?\n\nLIMITATIONS:\nThe limitations and societal impacts have already been discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of temporally correlated inverse problems by employing an image diffusion model. The authors introduce a technique termed Warped Diffusion, which incorporates an equivariance self-guidance mechanism. This mechanism ensures that the generated frames maintain consistency when subjected to warping transformations. The paper demonstrates the application of this technique in the context of video inpainting and video super-resolution through a series of experiments.\n\nSTRENGTHS:\n1. The paper is presented in a structured manner, with illustrations that aid in understanding. The content is coherent and follows a logical sequence.\n\n2. The authors offer an analysis of noise warping and equivariance self-guidance, providing both intuitive explanations and theoretical underpinnings.\n\n3. 3. The experiments are extensive to support the claim.\n\nWEAKNESSES:\n1. The authors report the noise warping speed at Line 357. We recommend that the authors also include the complete inference time required to process an entire video, which would provide a more comprehensive understanding of the method's efficiency.\n\n2. We suggest that the authors provide more video results to further demonstrate the capabilities and limitations of the proposed method in various scenarios.\n\n3. It is observed that the inpainted regions in the video inpainting examples remain static, such as the cat in the video not exhibiting motion. All apparent motion seems to be attributed to camera movements, with no optical flow present within the inpainted regions. We would appreciate an explanation for this phenomenon and its implications on the video inpainting process.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations and negative societal impact in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper follows a previous work on noise-warping methods for video inverse problem. The authors proposed instead of giving temporally consistent noise maps to the DM, using a continuous function space representation for DM directly can serve more complex spatial transformations and results in better temporal consistency in the generated videos. They demonstrated the proposed method on video inpainting and super-resolution.\n\nSTRENGTHS:\nThe proposed method is novel in terms of using Gaussian processes to handle the noise function so that no mapping is required for the noise. The authors derived the equivariance for deformation and by satisfying this the gaussian process is used to guide the generation to be consistent under the warping transformation. To avoid additional training, a sampling mechanism was proposed to ensure this satisfaction at inference time.\n\nWEAKNESSES:\nThe key statement in the paper is the importance of the equivariance of the DM to achieve temporal consistancy. The authors argued that the previous work \"How I Warped Your Noise\" performed badly when the DM is not equivariant. However, there are no explicit examples or solid proof on this statement. The proposed method for noise warping is more like offering another perspective than showing the failing reasons of the other work.\n\nFurthermore, for super-resolution experiments, the authors did not provide comparison against \"How I warped your noise\". Even though they elaborated that the authors of \"How I warped your noise\" acknowledged to them that the work does not result in temporal consistency, I believe it is still better to include the comparison given how many super-resolution cases were presented in that work. If they don't provide the code, the authors may consider modify the one they have for the inpainting experiments and list out the comparison for more convincing results.\n\nQUESTIONS:\n1. It is not clear to me why the function space diffusion models need to be equivariant with respect to the underlying spatial transformations. I can imagine the necessity, but in your derivation, I don't see the proof that you stated you provide. Can you provide explicit examples or solid proof on this statement?\n2. Why in fig 3 how i warped your noise seems to be in the second place in terms of error, but in table 2, the method seems not to outperform any other baselines?\n\nLIMITATIONS:\nThe authors did not address any limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-guidance equivariance approach using the Image Diffusion model for generating temporally consistent videos. In previous work HOW I WARPED YOUR NOISE[1], noise is warped across frames to ensure the noise maps are temporally consistent. However, temporally consistent noise maps do not guarantee temporally consistent output. In this work, the authors address this issue. The proposed self-guidance equivariance enforces that the generated frames remain consistent under the warping transformation. The proposed method can generate more consistent results. In their experiments, the authors conducted comparisons on video inpainting and superresolution. It is worth noting that the proposed method outputs consistent content in the inpainting area across frames.\n\nSTRENGTHS:\n- Problem Setup\n  - The problem seems reasonable and important, especially in the case of video inpainting. Compared to other video applications, any temporary inconsistent case can create a dramatic failure case. See Figure 1. \n\n- Temporal consistent results. \n    - In this work, the authors show how the proposed equivariance self-guidance improves video inpainting results and superresolution.\n\n- More efficient noise warping.\n   - Compared to the previous work [How I Warped Your Noise], this work improves the inference time by up to 16 times.\n\nWEAKNESSES:\n- Better illustration and explanation of how the `self-guidance Equivariance` works.\n  The main novelty of the paper is the self-guidance equivariance, which improves performance compared to previous work, HOW I WARPED YOUR NOISE [1]. However, it is not easy to understand how it works in Figure 2. The current Figure 2 only shows its role and the input and output in the overall system but does not describe its function. Besides, this figure seems to provide more information about how to use optical flow to guide the warping process of noise.\n\n- Inpainting examples and usefulness?\n  Do the warped noise and self-guidance Equivariance work for generating \"moving objects\" in video inpainting? According to the formulation and the generation results, it seems it will favor generating \"still objects\" in the video inpainting case because it will easily meet the \"warped\" mechanism.\n\n- Experiment Design.\n  Experiments in Section 4.1 are not needed (or do not provide a useful argument) in the paper. Previous works have shown that warped noise is better than independent noise across the frames in a video. The only contribution here is that the fine-tuned version performs better than the un-fine-tuned version. The finding seems unrelated to the paper or provides quite limited information.\n\nQUESTIONS:\n- A better way to illustrate how the proposed Equivariance Self-Guidance works.\n  - A proper figure would be better instead of just formulations.\n\n- See Point 2 in Weakness; can the proposed method handle cases where the object is moving or has different actions in the video inpainting case instead of being still?\n\nLIMITATIONS:\nFor the limitations, the authors mention that in some situations, the work may fail. It would be helpful to also provide the corresponding failing examples, such as the second and third examples mentioned in Appendix F Limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}