Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models

Authors: giannis  daras
weili  nie
alexandros g dimakis
morteza  mardani
nvidia  nikola
b  kovachki
nvidia  arash
vahdat  nvidia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:53:53

Abstract: Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos as a sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only on **images** and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the underlying spatial transformations. To ensure temporal consistency, we introduce a simple post-hoc test-time guidance towards (self)-equivariant solutions. Our method allows us to deploy state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and $8\times$ video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results in the following URL: https://giannisdaras.github.io/warped_diffusion.github.io/.

Main paper: 
 training with correlated noise: 
 The first step is to assess the quality of the trained models. To do so, we take images from a test split of the COYO dataset, we corrupt them (either by masking or downsampling) and we measure the conditional performance of the trained models. We use a diverse set of metrics that are commonly used in the inverse problems literature: CLIP Text Score [62] , CLIP Image Score [62] , SSIM [85] , LPIPS [91] , MSE, Inception Score [76] and FID [37] . The first five metrics measure point-wise restoration performance. Inception Score measures the quality of the generated distribution (without an explicit reference distribution). Finally, FID measures restoration performance in a distributional sense, i.e. it measures how close is the distribution after restoration to the ground truth distribution.We report our results for the super-resolution and inpainting models in Table 1 . The main finding is that finetuning with correlated noise does not compromise performance, i.e. SDXL models finetuned with correlated noise perform on par with SDXL models that are trained with independent noise. Particularly for inpainting, the GP models slightly outperform models trained with independent noise across all metrics. We provide qualitative results for our all models in Figure 1 and in Appendix Figures 7, 8 .We remark that the advantages of using an initial distribution other than white noise have been explored in prior work [20, 6, 42] . Our new finding is that a model initially trained with white noise can be easily fine-tuned to work with correlated noise. To the best of our knowledge, ours is the first work that shows that Stable Diffusion XL can be fine-tuned to work with correlated noise.We underline that prior to fine-tuning Stable Diffusion XL produces unrealistic images when the sampling chain is initialized with correlated noise. Our experiments show that post-finetuning, the model can handle spatially correlated noise in the input without compromising performance. Our GP Warping mechanism requires models that can handle correlated noise. Hence, these fine-tunings are essential for the rest of the paper.  
functional video generation: 
 The basis of our approach, summarized in Figure 2 , is to structure the generative model so that it is equivariant with respect to spatial deformations and apply these deformations successively to the input noise. Each deformation effectively warps the noise and the equivariance guarantees that each output image will be similarly warped. By using an optical flow from a real video to define a sequence of such deformations, a new video can be generated. To introduce our method, we first conceptualize both images and noise as functions on a domain and the generator as a mapping between two function spaces. Each video frame can be seen as a single image, and an image as a discretization of a vector-valued function on a rectangular domain. Consider the domain as the 2-D unit square D = [0, 1] 2 , defining an image as a function f : D → R 3 . For each location x ∈ D, the value f (x) ∈ R 3 represents an RGB color. We assume images have infinite resolution. To formulate a model that generates such images, we must have a notion of a space containing all possible images. We'll use the separable Hilbert space H = L 2 (D; R 3 ), with pointwise formulas interpreted almost everywhere with respect to the Lebesgue measure.We assume that there exists a probability measure µ on H whose support is the set of photorealistic images and denote by η a known reference probability measure on H. In our case, η will be a Gaussian measure on H; for details, see Section 3.1. A generative model, or transport map, is then a mapping G : H → H such that the pushforward of η under G is µ which we denote as G ♯ η = µ. In particular, this implies that any random variable ξ ∼ η will satisfy G(ξ) ∼ µ. For diffusion models, G can be defined by the probability flow ODE; see Section 3.2.Given an image f 0 ∈ H, a video with n+1 ∈ N frames is the sequence of functions (f 0 , f 1 , . . . , f n ) ∈ H n+1 , where each subsequent function is obtained, at least partially, from the previous one by a deformation. Specifically, a sequence of bounded, injective maps (T j : D → D j ) n j=1 exists such thatEQUATIONFrame Index Frame of Vision Deformed Domain where D j := T j (D) and we assume that the sets D ∩ D j have positive Lebesgue measure. In video modeling, the sequence (T j ) n j=1 is usually referred to as the optical flow as it specifies how each pixel in the previous frame moves to the next frame. While the frames can also be conceptualized as a continuum in time, we work with a discrete set of frames for simplicity. We consider D to always represent our fixed frame of vision and we allow each T j to move pixels outside of this frame. Therefore (1) determines f j only on the set D ∩ D j which contains pixels that remain within our field of vision.Given our notion of a video and a generative model, we now describe how such a model can be used to generate new videos. Suppose we want to create a two-frame video given an initial frame f 0 ∈ H and a deformation map T 1 : D → D 1 . Assume we have a generative model G : H → H and an initial noise image ξ 0 ∈ H such that G(ξ 0 ) = f 0 . From definition, the new frame of our video isf 1 = f 0 • T -1 1 on D ∩ D 1 . If D ⊆ D 1 ,it might seem that our generative model is unnecessary. However, proceeding this way generates blurry and unrealistic videos.The primary issue is that, in practice, we don't have access to f 0 at an infinite resolution but only at a fixed, finite set of grid points E k = {x 1 , . . . , x k } ⊂ D. To determine f 1 on our grid points, we need the values of f 0 at the points T -11(E k ) = {T -1 1 (x 1 ), . . . , T -1 1 (x k )}. It's highly unlikely that E k = T -11 (E k ) for any realistic deformation. Thus, we must interpolate f 0 to T -11 (E k ), which usually leads to blurry results with standard methods. Furthermore, if D ̸ ⊆ D 1 , there will be regions where f 1 is not determined by f 0 and will need to be inpainted on the new visible domain. Therefore, for each frame, we must solve an interpolation and an inpainting problem: tasks for which generative models are well-suited.Suppose we have access to the noise function ξ 0 at infinite resolution, and its domain extends to all of R 2 ; we discuss both in Section 3.1. We can then define the new frame in our video by applying the generative model to the deformed noise:f 1 = G(ξ 0 • T -1 1 ). The deformed noise function ξ 0 • T -1 1gets its values from ξ 0 | D for points in D ∩ D 1 and from the extension of ξ 0 to R 2 for all other points where inpainting is needed. To ensure this definition is consistent with (1), G must be equivariant with respect to T -1 1 . Specifically, for all ξ ∈ supp(η) ⊆ H, we must haveEQUATIONAssuming (2), it follows from G(ξ 0 ) = f 0 , that f 1 (x) = G(ξ 1 | D )(x) = (f 0 • T -1 1 )(x)for all x ∈ D ∩ D 1 hence the pair (f 0 , f 1 ) is a valid 2 frame video according to the definition of Section 2.1. To generate a video with any number of frames, we simply iterate on this process with a given sequence of deformation maps. Enforcing (2) can be done directly by the architectural design, through training with various deformation maps, or, through a guidance process; see Section 3.2.It is common practice to train generative models assuming the reference measure η is Gaussian white noise. Specifically, a draw ξ ∼ η on the grid points E k = {x 1 , . . . , x k } ⊂ D is realized as ξ(x l ) = χ l for an i.i.d. sequence χ l ∼ N (0, 1) for l = 1, . . . , k. However, this approach is incompatible with our goal of having the generative model perform interpolation. For most deformations T encountered in practice, none of the points in T -1 (E k ) will match those in E k . Consequently, each new evaluation ξ T -1 (x l ) will be independent of the sequence {χ l } k l=1 , making ξ T -1 (E k ) appear as a new noise realization unrelated to ξ(E k ). This incompatibility arises because white noise processes are distributions, not regular functions, meaning realizations are almost surely not members of H [18] . [14] proposes a stochastic interpolation method to address this issue (see Appendix C for details and comparison). We generalize this idea and propose using generic Gaussian processes on H. 
method: warped diffusion: 
 In Section 1, we formulated the problem of video generation as the computation of a series of functions warped by an optical flow and proposed the use of a generative model for inpainting and interpolating the warped functions. The main challenges which remain are defining a functional noise process which can be evaluated continuously and a generative model which is equivariant with respect to warping. We propose to use Gaussian processes for our functional noise and a guidance procedure within the sampling step of a diffusion model to overcome these challenges.A Gaussian Process (GP) η is a probability measure on H completely specified by its mean element and covariance operator. For a mathematical introduction, see Appendix B. We identify Gaussian processes with positive-definite kernel functions κ : R 2 × R 2 → R. Recall that E k = {x 1 , . . . , x k } denotes the grid points where we know the values of an image f ∈ H. To realize a random function ξ ∼ η on these points, we sample the finite-dimensional multivariate Gaussian N (0, Q), where Q ∈ R k×k is the kernel matrix Q ij = κ(x i , x j ) for i, j = 1, . . . , k.Once sampled, given the fixed values ξ(E k ), ξ can be evaluated at any new point x * ∈ D by computing the conditional distribution ξ(x * ) | ξ(E k ) [65] . This approach allows us to realize random functional samples at infinite resolution through conditioning, thus resolving the interpolation problem. Furthermore, by ensuring the kernel κ is positive definite on a domain larger than D, we can consistently sample ξ outside of D, addressing the inpainting problem described in Section 2.2.For high-resolution images when k is large, working with the matrix Q can be computationally expensive. Instead, we propose using Random Fourier Features (RFF) to sample η, which amounts to a finite-dimensional projection of the function ξ ∼ η that converges in the limit of infinite features [63, 87] . We can approximate samples from a GP with a squared exponential kernel with length-scale parameter ϵ > 0 by ξ(x) = 2 J J j=1 w j cos ⟨z j , x⟩ + b j for i.i.d. sequences w j ∼ N (0, 1), z j ∼ N (0, ϵ -2 I 2 ), b j ∼ U (0, 2π) where J ∈ N is the number of features. RFF allows us access to ξ at infinite resolution on the entirety of the plane while also allowing for efficient computation.We will now focus on the generative model that needs to be equivariant to the noise transformations. Specifically, in this section, i) we introduce function space diffusion models, ii) we prove that if every prediction of the diffusion model is equivariant then the whole diffusion model sampling chain is equivariant to the underlying spatial transformations, and, iii) we describe equivariance self-guidance, our sampling technique for enforcing the equivariance assumption.For ease of notation, we will present everything for the case of unconditional video generation. However, our method seamlessly incorporates any addition conditioning information that may be available. If c 0 , . . . , c n ∈ R c is a sequence of known conditioning vectors then these can simply be passed into a conditional score model at the appropriate frame without any other change to our method; see Algorithm 1. Conditioning vectors could be, for example, low resolutions versions of a video or an original video with regions masked. In Section 4, we focus on such conditional tasks.Function Space Diffusion Models. Typically, diffusion models are trained with white noise. As explained in Section 2.3, a principled continuous evaluation of the noise requires a functional process. We briefly describe diffusion models in the context of sampling using the Gaussian processes of Section 3.1. We show in Section 4.1 (Table 1 ) that a model trained with white noise can be fine-tuned to GP noise without any loss in performance.While it is possible to formulate diffusion models on the infinite-dimensional space H e.g. [52] , we will proceed in the finite-dimensional case for ease of exposition. In particular, we will define the forward and backward process as a flow on a vector u ∈ R k , thinking of the entries as the values of a scalar function evaluated on the grid E k and recall that Q is the kernel matrix on E k .We consider forward processes of the form,EQUATIONwhere W t is a standard Wiener process on R k and σ is a scalar-valued, once differentiable function. This process results in conditional distributions p(u t |u 0 ) = N (u 0 , σ 2 (t)Q), see [46] . Let p(u t , t) denote the density of u t induced by (3) . Then the following backward in time ODE,EQUATIONstarted at u(τ ) distributed according to (3) has the same marginal distributions p(u t , t) as (3) on the interval [0, τ ]; see [46] . Approximating N (u 0 , σ 2 (τ )Q) by N (0, σ 2 (τ )Q), we may then define the generative model G by the mapping u(τ ) → u(0) with reference measure η = N (0, σ 2 (τ )Q).Solving (4) requires knowledge of the score ∇ u log p(u t , t). Instead of learning the score, we opt for directly learning the weighted score Q∇ u log p(u t , t). This design choice leads to faster sampling since we do not need to perform any expensive matrix multiplication with Q at inference time.A generalized version of Tweedie's formula (for proof see Appendix A.2) implies:EQUATIONWe approximate E[u 0 |u t ] with a neural network h θ by minimizing the denoising objective:EQUATION)Having a minimizer h θ of ( 6) gives us access to the weighted score Q∇ u log p(u t , t) via ( 5). We may then obtain an approximate solution to the map u(τ ) → u(0) by discretizing (4) in time. We consider Euler scheme updates given byEQUATIONstarted with u τ ∼ N (0, σ 2 (τ )Q) for some time step ∆t > 0.Equivariance for the Probability Flow ODE. Since the diffusion model works with discrete inputs, we need to introduce a discretization of (2) for the network. For a deformation T 1 , we define equivariance asEQUATIONwhich is obtained from composing both sides of (2) with T 1 . Note that ( 8) is valid only for pixels which stay within frame and we compute the l.h.s. with bilinear interpolation on the network output.The input to the network on the l.h.s. is computed with RFFs without any interpolation. Given this discrete equivariance is satisfied for every prediction of the network, it is straightforward to show that the whole diffusion model sampling chain will be equivariant. Indeed, the whole approximation to u(τ ) → u(0) is equivariant by the linearity of composition -for a full derivation, see Appendix A.3.Equivariance Self-Guidance. The condition ( 8) is rarely satisfied for deformations T 1 arising in practical settings. This is because either the conditional expectation E[u 0 |u t ] is not equivariant with respect to T -11or the neural network approximation has not fully captured it. If the underlying equivariance assumption breaks, methods that rely solely on noise warping for temporal consistency, e.g. [14] , will perform poorly. This is evident in challenging conditional tasks (see Figure 1 ).A potential solution is to directly train the network by adding (8) as a regularizer. However, this requires large amounts of video data from which to extract optical flows. Furthermore, by satisfying (8) over a large class of T 1 (s), the network may become less apt at satisfying (6) and lose its generative abilities. Therefore, we opt for guiding the model towards equivariant solutions at inference time.We first sample noise u (0) τ and generate the first frame following (7) , keeping the outputs of the network at each time step {h θ (u (0) t , t)}. To generative the next frame, we warp our noise u(1) τ = u (0) τ • T -1 1with RFFs and again follow (7) but this time using (8) as guidance. In particular, we take a gradient steps in the direction of the loss function |h θ (u(1) t , t)•T 1 -h θ (u (0) t , t)| 2 ,computed on the pixels that stay within frame. All frames can be generated by iterating this procedure as summarized in Algorithm 1 (and visualized in Figures 2, 9 ) which also shows how to use conditioning information. Guidance is typically used to solve inverse problems with diffusion models (e.g. see [15] ), but here the guidance is applied to align the model with its own past predictions. We emphasize that to compute the composition with T 1 above, we use bilinear interpolation on the network outputs but we never need to interpolate the network inputs since we can compute the warping via RFFs. Furthermore, since we are matching interpolated outputs to ones that are not interpolated, our output images remain sharp. This in contrast to directly using a discrete version of (2) which would suggest that we match network outputs to interpolated images, producing blurry results. 
experimental results: 
 For all our experiments, we use Stable Diffusion XL [60] (SDXL) as our base image diffusion model. We start by finetuning SDXL on conditional tasks. We choose super-resolution and inpainting as the tasks of interest since they are both commonly used in the inverse problems literature and they represent two distinct scenarios: in super-resolution, the input condition is strong and in inpainting, the model needs to generate new content. For super-resolution, we choose a downsampling factor of 8. For inpainting, we create masks of different shapes at random, following the work of [57] . During the finetuning, we train the model to predict the uncorrupted image given the following inputs: i) the encoding of the noised image, ii) the noise level, and, iii) the encoding of the corrupted (downsampled/masked) image. To condition on the corrupted observation, we concatenate the measurements across the channel dimension. We train models with and without correlated noise on the COYO dataset [12] for 100k steps. We show realizations of independent and correlated noise in Figure 6 . Additional implementation details are in Section F.2, including the parameters for the GP introduced in Section 3.1.Require: Conditioning vectors {cj} n j=0 , Step ∆t, Time τ , Schedule σ(t), Model h θ , Guidance Strength λ.1: {Tj, T -1 j } n j=1 ← compute_optical_flow({cj} n j=0 ) 2: u (0) τ ∼ GP using RFFs in Section 3.1▷ Fresh noise sample for the first frame 3: Compute trajectory {u (0) t } using (7) ▷ Sample first frame 4: for j ← 1 to n do 5:u (j) τ ← u (j-1) τ • T -1 jusing RFFs ▷ Warp noise from previous frame 6:t ← τ 7:while t > 0 do 8: u(j) t-∆t ← u (j) t -∆t σ(t) σ(t) h θ (u (j) t , t, cj) -u (j) t ▷ Take Euler step 9: e (j) t ← h θ (u (j) t , t, cj) • Tj -h θ (u (j-1) t , t, cj-1) 2 ▷ Compute warping error 10: u (j) t-∆t ← u (j) t-∆t -λ √ e t ∇ue (j) t ▷ Equivariance self guidance 11: t ← t -∆t 12:end while 13: end for 14: return video {u(j) 0 } n j=0 4.In the previous experiments, we measured the restoration performance of the trained models for a single image and we established that models trained with correlated noise perform on par (or even outperform) models trained with independent noise. The next step is to measure the temporal behavior of the models, i.e. how well they work for videos.Noise Warping baselines. As explained in Section 1, to apply image diffusion models to videos, we need to transform the noise as we move from one frame to the next. We consider the following noise-warping baselines that were used in [14] : Fixed noise uses the same noise across all the frames. Resample noise samples a new noise for each new frame. Nearest Neighbor uses the noise of the nearest location in the grid to evaluate the noise at the location that is not on the regular grid E k .Bilinear Interpolation interpolates the values of the noise bilinearly in the neighboring locations that lie on the grid. How I Warped Your Noise [14] is the state-of-the-art method for solving temporally correlated inverse problems with image diffusion models. It warps the noise by using auxiliary high-resolution noise maps (see our intro, related work section, and Section C). Our GP Noise Warping warps the input noise by resampling the Gaussian process in the mapped locations. We note that the Fixed Noise, Resample Noise, and Nearest Neighbor noise warping methods can be applied to models that are trained with either independent noise or correlated noise coming from a GP. For all the experiments, we also include our proposed method, Warped Diffusion that uses GP Noise Warping and Equivariance Self-Guidance (see Algorithm 1 for a reference implementation).Video Evaluation Metrics. We follow the evaluation methodology of the "How I Warped Your Noise" paper [14] . Specifically, we want to measure two different aspects of our method: i) average restoration performance across frames, ii) temporal consistency. For i), we measure the average of all the previously reported metrics (FID, Inception, CLIP Image/Text score, SSIM, LPIPS and MSE) across the frames. For ii), we measure the self-warping error, i.e. how consistent are the model's predictions across time. The warping error can be computed in either pixel or latent space and also with respect to the first generated frame or the previously generated frame, totaling 4 warping errors.To warm up, we start with videos that are synthetically generated by 2-D shifting of a single image, as in Figure 1 . To further simplify the setup, we consider the easy case of shifting the current frame by an integer amount of pixels with each new frame. For 2-D translations by an integer amount of pixels, the Nearest Neighbor, Bilinear Interpolation, How I Warped Your Noise and GP Noise Warping methods they become essentially the same since we always evaluate the noise distribution on points in the grid E k . Hence, the only difference is whether we apply these methods to white noise or to GPs.Figure 1 (Row 2) shows that the How I Warped Your Noise baseline produces temporally inconsistent results as we shift the masked input image. Even though all the inpaintings are of high quality, the baseline results are temporally inconsistent. Instead, our Warped Diffusion method produces temporally consistent results since it enforces equivariance by design. Since the How I Warped Your Noise warping mechanism and GP coincide here, the benefit strictly comes from enforcing the equivariance property. In fact, one could get the same results for the How I Warped Your Noise method by penalizing for equivariance at inference time.We present quantitative results regarding temporal consistency in Figure 3 (and additional results in Figure 4 in the Appendix). As shown in the Figure, the fixed noise and the resample noise baselines perform the worst w.r.t. the temporal consistency both in latent and pixel space. The warping error of the Resample baseline is almost constant across frames as expected, while the warping error of the Fixed Noise increases with time. Both the How I Warped Your Noise method and our GP warping framework significantly improve the baselines. Yet, they still have significant temporal inconsistencies as evidenced by the results in Figure 1 and the supplemental videos. The two methods perform on par on this task since they are essentially the same when it comes to integer shifts: the only difference is that GP Noise Warping is applied to correlated noise coming from a GP. The remaining temporal errors are not an artifact of the noise warping mechanism but they are due to the fact that the model itself is not equivariant w.r.t. the underlying transformation. The warping errors essentially disappear when we apply Equivariance Self Guidance. As shown in Figure 3 , our method, Warped Diffusion, achieves almost 0 warping error (1e-4 mean pixel error with respect to the first frame to be precise) since it is enforcing equivariance by design.The only remaining question is whether Warped Diffusion maintains good restoration performance.To answer this, we measure mean restoration performance across frames for the aforementioned metrics. We report our results in Table 2 , including the mean warping error with respect to the first frame. As shown, Warped Diffusion maintains high performance across all the considered metrics while being significantly superior in terms of temporal consistency. The conclusion is that all the other noise warping baselines, including the previous state-of-the-art How I Warped Your Noise paper [14] , perform poorly in terms of temporal consistency since they rely on the assumption that the network is equivariant. Even for simple temporal correlations such as integer movement in the 2-D space, this assumption is false for the challenging inpainting task. Warped Diffusion is the only method that achieves temporal consistency while it still manages to maintain high reconstruction performance. We finally remark that our sampling algorithm enforces equivariance in the latent space. Yet, the warping errors are negligible in the pixel space as well. Our finding is that improving latent space equivariance translates to improvements in pixel space equivariance. The authors of [14] also find that "the VAE decoder is translationally equivariant in a discrete way".We proceed to evaluate our method on realistic videos. We measure performance on 600 captioned videos from the FETV [55] dataset. Since baseline inpainting methods fail even for very simple temporal transformations, we focus on 8× super-resolution for our comparisons on FETV.For our video results, we could not provide comparisons with the How I Warped Your Noise paper. At the time of this writing, there was no available reference implementation as we confirmed with the authors by direct communication. In any case, the authors acknowledge as a limitation of their work that their proposed method has "limited impact on temporal coherency" when applied to latent models and that "all the noise schemes produce temporally inconsistent results" [14] . Once again, we attribute this to the non-equivariance of the denoiser, which we mitigate with our guidance algorithm.We proceed to evaluate our method and the baselines with respect to temporal consistency and mean restoration performance across frames, as we did for our inpainting experiments. We present our results in Table 3 and additional results in Figures 5, 8 of the Appendix and in the following URL as videos: https://giannisdaras.github.io/warped_diffusion.github.io/. As shown in Table 3 , there is a trade-off between temporal consistency and restoration performance. Methods that perform better in terms of temporal consistency often have significantly worse performance across the other metrics. Our Warped Diffusion achieves a sweet spot: it has the lowest warping error by a large margin and it still maintains competitive performance across all the other metrics. On the contrary, methods that are based solely on noise warping, such as GP Warping and the simple interpolation methods, lead to significant performance deterioration for a small improvement in temporal consistency.Noise Warping Speed. We measure the time needed for a single noise warping. Our GP Warping mechanism takes 39ms per frame Wall Clock time, to produce the warping at 1024 × 1024 resolution. This is 16× faster than the reported 629ms number in [14] . If we use batch parallelization, our method generates 1000 noise warpings in just 46ms (at the expense of extra memory).No Warping? A natural question is whether we can omit completely the noise warping scheme since equivariance is forced at inference time. We ran some preliminary experiments for super- resolution on real-videos and we found that omitting the warping significantly deteriorates the results when the number of sampling steps is low. We found that increasing the number of sampling steps makes the effect of the initial noise warping less significant, at the cost of increased sampling time. 
limitations: 
 Our method has several limitations. First, the guidance term increases the sampling time, as detailed in the Appendix, Section F.3. For reference, processing a 2-second video takes roughly 5 minutes on a single A-100 GPU. Second, even though in our experiments we observed a monotonic relation between the warping error in latent space and warping error in pixel space, it is possible that for some transformations the decoder of a Latent Diffusion Model might not be equivariant. We noticed that this is a common failure for text rendering, e.g. in this latent video the model seems to be equivariant, but in the pixel video it is not. Third, the success of our method depends on the quality of the flow estimation -inconsistent flow estimation between frames will lead to flickering artifacts. For real videos, there might be occlusions and the estimation of the flow map can be noisy. We observed that in such cases our method fails, especially for challenging tasks such as video inpainting. The correlations obtained by following the optical flow field obtained from real videos might lead to a distribution shift compared to the training distribution. For such extreme deformations, our method produces correlation artifacts. This has been observed in prior work (see this video), but it also appears in our setting (e.g. see this video). Finally, our method cannot work in a zero-shot manner since it requires a model that is trained with correlated noise. 
conclusions: 
 Warped Diffusion is a novel framework for solving temporally correlated inverse problems with image diffusion models. It leverages a noise warping scheme based on Gaussian processes to propagate noise maps and it ensures equivariant generation through an efficient equivariance self-guidance technique. We extensively validated Warped Diffusion on temporally coherent inpainting and superresolution, where our approach outperforms relevant baselines both quantitatively and qualitatively. Importantly, in contrast to previous work [14] , our method can be applied seamlessly also to latent diffusion models, including state-of-the-art text-to-image models like SDXL [60] . 
: 
 Diffusion models (DMs) [79, 39, 82] can synthesize photorealistic imagery [73, 64, 66, 5, 60, 26] . They can be conditioned easily, through explicit training or guidance [25, 41] , and have also been widely used to solve inverse problems [19, 86, 15, 16, 80, 84, 47, 56] , in particular for image processing applications like inpainting and super-resolution [40, 72, 74, 66] .How do these methods extend to video processing and solving inverse problems on videos? Although video DMs are seeing rapid progress [38, 78, 9, 29, 8, 30, 7, 11] , general textto-video synthesis has not yet reached the level of robustness and expressivity comparable to modern image models. Moreover, no state-of-the-art video generative models are publicly available [11] , and most video DMs are computationally expensive. To circumvent these challenges, a natural research direction is to leverage existing, powerful image generative models to solve video inverse problems.Naively applying image DMs to videos in a frame-wise manner violates temporal consistency. Previous works alleviate the problem by fine-tuning on video data or by warping the networks' features, using, for instance, temporal or cross-frame attention layers [88, 54, 13, 51, 61, 90, 92, 34, 33] . However, these methods are usually designed specifically for high-level text-driven editing or stylization and are typically not directly applicable to general inverse problems. Moreover, without training on diverse video data they often cannot maintain high frequency information across frames. For a detailed discussion of the related works, we refer the reader to Section E in the Appendix.The recent novel work, "How I Warped Your Noise" [14] , proposes noise warping to achieve temporal consistency in generated videos by changing appropriately the input noise to the diffusion model. Videos can be thought of as image frames subject to spatial transformations. An object may move according to a translation; complex and general transformations can be described by motion vectors on the pixels defined through optical flow [27] . It is these transformations that define how the noise maps need to be warped and transformed. In [14] , temporally consistent noise maps are given as input to the DM's denoiser, with the underlying assumption that temporally consistent inputs induce temporally consistent network outputs. In this paper, we argue that this assumption only holds true if the utilized image DM is equivariant with respect to the spatial warping transformations. However, as we show in this work, the network is not necessarily equivariant because i) the conditional expectation modeled by the DM may not be equivariant, and, ii) more importantly, a free-form neural network, as used in typical DMs, will not learn a perfectly equivariant function. When the equivariance assumption is violated, the method proposed in [14] achieves poor results. This is typically the case for challenging conditional tasks (see Figure 1 ) or when modeling complex distributions. Particularly, [14] finds that the proposed method has "limited impact on temporal coherency" when applied to latent diffusion models and that "all the noise schemes produce temporally inconsistent results".We introduce a new framework, dubbed Warped Diffusion, for the rigorous application of image DMs to video inverse problems. We employ a continuous function space perspective to DMs [52, 59, 28, 35] that naturally allows noise warping for arbitrarily complex spatial transformations. Our method generalizes the warping scheme of [14] and does not require any auxiliary high-resolution noise maps.To achieve equivariance, we propose equivariance self-guidance, a novel sampling mechanism that enforces that the generated frames are consistent under the warping transformation. Our inference time approach elegantly circumvents the need for additional training. This unlocks the use of existing large DMs in a fully equivariant manner without further training, which may be prohibitive for a practitioner.We extensively validate our method on video inpainting and super-resolution. Super-resolution represents a situation with strong conditioning, while inpainting requires large-scale, temporally coherent synthesis of new content. Warped Diffusion outperforms previous methods quantitatively and qualitatively, and shows reduced flickering and texture sticking artifacts. Due to our equivariance guidance, our method can also be used with latent DMs, which is not possible with previous approaches. Virtually all existing state-of-the-art text-to-image generation systems are indeed latent DMs, like Stable Diffusion [66] . Hence, any inverse problem solving method must be readily usable with latent DMs. In fact, all our experiments utilize the state-of-the-art text-to-image latent DM SDXL [60] .  
 
 Appendix: To better understand (2), we consider the following example. We will assume that ξ : R 2 → R is a scalar-valued field (grayscale image) defined on the whole plane. Furthermore, we let G be given by a continuous convolution and T -1 1 be a translation. In particular, for all x ∈ R 2 ,G(ξ)(x) = R 2 κ(x -y)ξ(y) dy, T -1 1 (x) = x -afor some compactly supported kernel κ : R 2 → R and a direction a ∈ R 2 . We then haveG ξ • T -1 1 x = R 2 κ(x -y)ξ(y -a) dy = R 2 κ (x -a) -y ξ y dy = G ξ T -1 1 (x)by the change of variables formula. This shows that G is equivariant to all translations. This is a well-known property of the convolution and, in particular, it shows that convolutional neural networks are translation equivariant, noting that pointwise non-linearities will preserve this property. This example shows that a model can be equivariant with respect to certain deformations by architectural design. However, in realstic video modeling, optical flows are not know explicitly and can only be approximated numerically. It is therefore natural to instead build-in approximate equivariance into a model instead of enforcing it directly in the architecture. Our guidance procedure in Section 3.2 is an example such an approximate form of equivariance.In Section 3.2, we show a diffusion model can be trained and sampled from using Gaussian process noise instead of white noise. Our result depends on the following lemma which is a simple generalization of Tweedie's formula. Lemma A.1. Let x be a random variable with positive density p x ∈ C 1 (R k ). Let σ > 0 and z ∼ N (0, Q) for some positive definite matrix Q ∈ R k×k and assume that x ⊥ z. Define the random variable y = x + σz and let p y ∈ C ∞ (R k ) be the density of y. It holds that∇ y log p y (y) = 1 σ 2 Q -1 E[x|y] -y .Proof. First note that by the chain rule,∇ y log p y (y) = 1 p y (y) ∇ y p y (y) = 1 p y (y) ∇ y R k p(y, x) dxwhere p(y, x) denotes the joint density of (y, x). Let p(y|x) denote the Gaussian density of the conditional y|x. Sincep x ∈ C 1 (R k ), ∇ y R k p(y, x) dx = R k ∇ y p(y|x)p x (x) dx.Therefore, by the chain rule, ∇ y log p y (y) = 1 p y (y) R k p(y|x)p x (x)∇ y log p(y|x) dx.Since p(y|x) is the density of N (x, σ 2 Q), a direct calculations shows that∇ y log p(y|x) = 1 σ 2 Q -1x -y . Furthermore, Bayes' theorem implies p(y|x)p x (x) = p(x|y)p y (y). Therefore,∇ y log p y (y) = 1 σ 2 R k Q -1 x -y p(x|y) dx = 1 σ 2 Q -1 E[x|y] -y as desired.In Section 3.2, we claim that if the score network h θ is equivarient with respect to a deformation T -1 , then the Euler scheme approximation of the map u(τ ) → u(0) is equivarient with respect to T -1 . It is easy to see that this results holds so long as it holds for the single step u t → u t-∆t defined by (7) . We will assume that h θ safisfies (8) written ash θ (u t • T -1 , t) = h θ (u t , t) • T -1 .We make sense of this equation by using RFF to define u t as a function on the plane and similarly bilinear interpolation to define h θ (u t , t) as a function. It follows by linearity of composition thatu t-∆t • T -1 = u t • T -1 -∆t σ(t) σ(t) h θ (u t , t) • T -1 -u t • T -1 = u t • T -1 -∆t σ(t) σ(t) h θ (u t • T -1 , t) -u t • T -1which is the requisite equivariance of the map u t → u t-∆t .A probability measure η on H is called Gaussian if there exists an element m ∈ H and a self-adjoint, non-negative, trace-class operator Q : H → H such that, for all h, h ′ ∈ H,⟨h, m⟩ = H ⟨h, f ⟩ dη(f ), ⟨Qh, h ′ ⟩ = H ⟨h, f -m⟩⟨h ′ , f -m⟩ dη(f ),where ⟨•, •⟩ denotes the inner product on H. The element m is called the mean while the operator Q is called the covariance. It is immediate from this definition that white noise is not included since the identity operator is not trace-class on any infinite dimensional space. This definition ensures that any realization of a random variable ξ ∼ η is almost surely an element of H. When the domain D of the elements of H is a subset of the real line, η is often called a Gaussian process. We continue to use this terminology even when D is a subset of a higher dimensional space i.e. R 2 but remark that the nomenclature Gaussian random field is sometimes preferred.Since we working on a separable space, each such field on H has associated to it a unique reproducing kernel Hilbert space [18, Theorem 2.9] which is associated to a unique positive definite kernel [4] .In particular, there exists a positive definite function κ : D × D → R for which Q is its associated integral operator. It follows that a Gaussian process can be uniquely identified with a positive definite kernel. Sampling and conditioning this process can then be accomplished via the kernel matrix.To make this explicit, suppose that X = {x 1 , . . . , x n } ⊂ D and Y = {y 1 , . . . , y m } ⊂ D are two sets of points in D. We will slightly abuse notation and writeQ(X, Y ) ij := κ(x i , y j ), i = 1, . . . , n and j = 1, . . . , mfor the kernel matrix between X and Y and similarly Q(Y, X), Q(X, X), Q(Y, Y ). Suppose that ξ ∼ η is a random variable from the Gaussian process with kernel κ and mean zero. To sample a realization of ξ on the points X, we sample the finite dimensional Gaussian N 0, Q(X, X) . This can be written asξ(X) = Q(X, X) 1/2 Zwhere Z ∼ N (0, I n ). Suppose now that the points in Y are distinct from those in X and we want to sample ξ on Y given the realization ξ(X). This can be done by conditioning [65] ξ(Y )|ξ(X) ∼ N Q(Y, X)Q(X, X) -1 ξ(X), Q(Y, Y ) -Q(Y, X)Q(X, X) -1 Q(X, Y ) .While the above formulas fully characterize sampling ξ, working with them can be computationally burdensome. It is therefore of interest to consider a different viewpoint on Gaussian processes, in particular, through the Karhunen-Loève expansion. The spectral theorem implies that Q possesses a full set of eigenfunctions Q j ϕ j = λ j ϕ j for j = 1, 2, . . . with some decaying sequence of eigenvalues λ 1 ≥ λ 2 ≥ . . . . The random variable ξ ∼ N (0, Q) can be written asξ = ∞ j=1 λ j χ j ϕ jwhere χ j ∼ N (0, 1) is an i.i.d. sequence and the right hand side sum converges almost surely in the norm of H [18] . By truncating this sum to a finite number of terms, computing realizations of ξ becomes much more computationally manageable. This inspires the random features approach to Gaussian processes which is the basis of our computational method; for precise details, see [65, 63, 87] .We show in Section 2.3 that a white noise process is not compatible with the idea of using a generative model to interpolate deformed functions. A potential way of dealing with this issue is to treat the original realizations of the white noise ξ(E k ) as the fixed nodal points of a function ξ and obtain the rest of the values via interpolation. It is shown in [14] that common forms of interpolation yield a conditional distribution ξ T -1 (E k ) |ξ(E k ) that is too dissimilar from the training distribution N (0, I k ) and thus the generative model produces blurry or disfigured images.Therefore [14] proposes a stochastic interpolation method which has the property that, for a new point x * ̸ ∈ E k , the distribution of ξ(x * ) marginalized over the joint distribution ξ(E k ), ξ(x * ) follows N (0, 1). This is most easily seen in one spatial dimension with k = 2 points. Suppose that D = [0, 1] and let a, b ∼ N (0, 1) be two independent random variables. Consider a Gaussian process on D with kernel function κ(x, y) = 1 -|x -y| and suppose that ξ is distributed according to this GP conditioned on ξ(0) = a and ξ(1) = b. A straightforward calculation shows that, for any x * ∈ (0, 1),ξ(x * ) = (1 -x * )a + x * b + 2x * (1 -x * )zfor z ∼ N (0, 1) independent of (a, b). This is simply the Brownian bridge connecting a to b.Remarkably, the marginal distribution of ξ(x * ) over the joint (ξ(x * ), a, b) is N (0, 1) independently of x * . However, the conditional distribution isξ(x * )|a, b = N (1 -x * )a + x * b, 2x * (1 -x * )which is not N (0, 1) for all x * ∈ (0, 1). In [14, Section 2.2], it is proposed that such Brownian bridges are used between any two pair of pixels, yielding a stochastic interpolation method given by a sequence of such independent GPs. However, from the point of view of using a generative model that is pre-trained on N (0, I k ), it is not of interest that the marginal distribution of ξ(x * ) is N (0, 1) but rather that the conditional ξ(x * )|a, b is N (0, 1). As we have seen, this is not the case for the method of [14] and, in fact, it will only ever be the case for white noise processes as discussed in Section 2.3. Therefore, no matter what method is used, there will always be a distribution shift to the model input induced by the deformation T -1 . A well chosen noise process will simply try to minimize this shift as much a possible.The work [14] proposes to use diffusion models trained on discrete inputs distributed according to N (0, I k ) and computes conditional distributions ξ T -1 (E k ) |ξ(E k ) using the stochastic interpolation method described above, generalized to two dimensions. We, instead, propose to use a Gaussian process N (0, Q), as described in Section 3.1 and compute ξ T -1 (E k ) |ξ(E k ) by conditioning this process which amounts to simply evaluating the RFF projection. It is our numerical experience that this better preservers the qualitative properties of the input distribution for large deformations. We leave the exploration of a process best suited for this task as important future work.In this section, we provide additional results that did not fit in the main paper. We visualize the difference between independent noise and noise from our GP in Figure 6 . We present inpainting results from our SDXL inpainting model fine-tuned with GP noise in Figure 7 . We present superresolution results from our SDXL super-resolution model fine-tuned with GP noise in Figure 8 . We further present warping errors with respect to the previous frame in Figure 4 for the inpainting results and warping errors for super-resolution for real videos in Figure 5 . Finally, we present additional comparisons for super-resolution in Figure 10 .Our work is primarily related to three recent lines of research about the utility of diffusion models in inverse problems, video editing, and equivariance in function space diffusion models as elaborated below.Diffusion Models for Inverse Problems. Diffusion models have been recently received widespread adoption for solving inverse problems in various domains. Diffusion models can solve inverse problems in a few different ways. A simple way is to train or finetune a conditional diffusion model for each specific task to learn the conditional distribution from the degraded data distribution to the clean data distribution [71, 53, 77] . Some popular examples include SR3 [75] and inpainting stable diffusion [67] . We leverage stable diffusion inpainting in the present work. While successful, they however need to be trained (or finetuned) separately for each individual task that is computationally complex. Also, they are not robust to out of distribution data. To mitigate these challenges, plugand-play methods have been introduced that utilize a single foundation diffusion model (e.g., stable diffusion) as a (rich) prior to solve many inverse problems at once [44, 70, 15, 47] . The crux of this approach is to modify the sampling post-hoc by either: (i) add guidance to the score function of diffusion models as in [15, 81] ; (ii) approximated projection onto the measurement subspace at each diffusion step [16, 47] or, (iii) use regularization by denoising via optimization [56, 32] . In this work we adopt the guidance-based approach to impose equivariance for the score function. All these methods have been applied for 2D images. For video inverse problems, the problem is more challenging due to temporal consistency. There are some efforts to leverage diffusion models for example for text-to-video superresolution or inpainting; see e.g., [69, 31, 93] . However, there is no systematic framework yet based on 2D diffusion models to solve generic video inverse problems in a temporally consistent manner. This is essentially the focus of our work. Finally, we remark that recent work [22, 21, 23, 2, 48, 1] has shown that it is even possible to train diffusion models to solve inverse problems without ever seeing clean images from the distribution of interest.Video Editing with Image Diffusion Models. Due to the lack of full-fledged pre-trained text-to-video diffusion models, many works focus on video editing (or video-to-video translation) using text-toimage diffusion models. One line of research has proposed to fine-tune the image diffusion model on a single text-video pair and generate novel videos that represent the edits at inference [88, 54, 92] . Specifically, Tune-A-Video [88] proposed a cross-frame attention mechanism and an efficient oneshot tuning strategy. Video-P2P [54] further improved the video inversion performance by optimizing a shared unconditional embedding for all frames. EI 2 [92] refined the temporal modules to resolve semantic disparity and temporal inconsistency of video editing. However, the fine-tuning process over the input video makes the editing less efficient. Another line of research has developed various training-free methods for efficient video editing, which mostly rely on the cross-frame attention and latent fusion for maintaining temporal consistency [13, 51, 61, 90] . In particular, Text2Video-Zero [51] encoded the motion dynamics in latent noises through a noise wrapping. FateZero [61] fused the attention features with a blending mask obtained by the source prompt's cross-attention map. Pix2Video [13] proposed to progressively propagate the changes to the future frames via self-attention feature injection. Rerender-A-Video [90] proposed hierarchical cross-frame constraints with the optical flow for improved temporal consistency.Recently, several works [52, 49, 50] have extended diffusion models to function data. However, these methods primarily focus on theoretical developments and have been examined on simplistic datasets such as time series, Navier-Stokes solutions, or hand-written digits. This paper can be considered one of the first successful applications of function-space diffusion models to natural image datasets. Our work is also related to the equivariant diffusion models which have been extensively explored in scientific applications such as molecule and protein interaction and generation applications [43, 3, 89, 17, 45] . However, equivariant diffusion models for image generation are less explored, primarily because guaranteeing equivariance (for example with respect to translation, rotation, or rescaling) in commonly used diffusion architectures such as U-Net [68, 39] or Transformer [58, 36] models is challenging.Diffusion models trained with correlated noise. Ours is not the first work to train diffusion models with a prior other than white noise. The authors of [20, 42] show how to train diffusion models with blurring corruption, leading to a blurred terminal distribution. Several other works have shown how to generalize diffusion models to find mappings between arbitrary input-output distributions, including [6, 10, 24] . One new finding in our work is that it is possible to start with a state-of-the-art model trained with white noise and fine-tune it easily to handle correlated noise. This allows us to convert vanilla diffusion models to Function Space Diffusion models by training them with noise sampled from Gaussian Processes. For more details, we refer the reader to Section 3.1.We use the RAFT model to predict the optical flows [83] . The optical flows can be computed with respect to the first frame or between subsequent frames. We find that the optical flow estimation is much better between subsequent frames and we use subsequent transformations to find the position in the original frame, whenever possible.Since we are working with Latent Diffusion Models, all the warping happens in a lower-dimensional space. Fortunately, as observed in numerous prior works, including [57] , there is a geometric correspondence between pixel blocks and latent locations, i.e. pixel blocks are mapped to specific locations in latent space. This allows us to extract the flows from the input frames and convert them to optical flows for our latent vectors. Alternatively, one nat first map to latent space and then compute the optical flow there. We did not pursue this approach since we rely on a deep learning method for the flow-estimation and the underlying model has been trained on natural images.To fine-tune SDXL in conditional tasks, we use the reference implementation found in the following link: https://github.com/huggingface/diffusers/pull/6592. The reference implementation finetunes SDXL on the inpainting task, however, it is straightforward to adapt it to other conditional tasks, such as super-resolution. As mentioned in the paper, we train all our models for 100, 000 steps. We use the following training hyperparameters:• Training resolution: 1024 × 1024.• Batch size: 64.• Latent resolution: 128.a general rule of thumb, we found that setting the length scale to be ϵ := 2 π•resolution leads to noise realizations that can be used to easily fine-tune Stable Diffusion XL.We train all our models on 16 A100 GPUs on a SLURM-based cluster. The fine-tuning of the SDXL model on conditional tasks (super-resolution, inpainting) with correlated noise for 100k steps takes roughly 24 hours.Sampling guidance for equivariance increases the generation time for two reasons: i) we need to run more steps in order to make it effective and, ii) each step is more expensive since we need to perform an additional backpropagation. For our experiments, we use 50 steps instead of 25 steps that we use for unconditional sampling. Further, without guidance, we get 4.32 iterations per second on a single A100 GPU while with guidance we obtain 1.62 iterations per second.The other hyperparameter used in sampling is the guidance strength, see Algorithm 1. For λ = 0, there is no guidance and the method just becomes GP Noise Warping. For higher λ the gradient from the warping guidance becomes stronger. In our experiments, we found the value λ = 1 to perform the best. This is consistent with the choice of λ in the Diffusion Posterior Sampling [15] paper which uses a guidance term to apply diffusion models for general inverse problems.We perform all our sampling experiments on a single A-100 GPU. Without sampling guidance, it takes roughly 20 seconds to generate a single frame. We measure the performance of our method and the baselines on 2 second videos consisting of 16 frames.Our method allows the use of image diffusion models to solve video inverse problems. There are both positive and negative societal implications of such a method. On the positive side, our method does not require training of video models which is typically expensive and contributes to increasing the AI carbon footprint. Further, democratizes access to video editing tools. The average practitioner can now leverage state-of-the-art image models to solve video inverse problems. To illustrate the effectiveness of our method, we trained powerful text-conditioned inpainting models that work on arbitrary images from the web. On the negative side, these models can be used for adversarial image and video editing. Further, our method can be used for the generation of deepfakes. 


Decision: All the 4 reviewers recommend acceptance. The authors should add discussions on failure cases, improve the organization and the presentation of the manuscript, and include the additional results and clarifications into the final copy.

Reviews: review: paper_summary: paper_summary: This paper proposes a self-guidance equivariance approach using the Image Diffusion model for generating temporally consistent videos. In previous work HOW I WARPED YOUR NOISE[1], noise is warped across frames to ensure the noise maps are temporally consistent. However, temporally consistent noise maps do not guarantee temporally consistent output. In this work, the authors address this issue. The proposed self-guidance equivariance enforces that the generated frames remain consistent under the warping transformation. The proposed method can generate more consistent results. In their experiments, the authors conducted comparisons on video inpainting and superresolution. It is worth noting that the proposed method outputs consistent content in the inpainting area across frames.
strength_weakness: strength_weakness: - Better illustration and explanation of how the `self-guidance Equivariance` works.
  The main novelty of the paper is the self-guidance equivariance, which improves performance compared to previous work, HOW I WARPED YOUR NOISE [1]. However, it is not easy to understand how it works in Figure 2. The current Figure 2 only shows its role and the input and output in the overall system but does not describe its function. Besides, this figure seems to provide more information about how to use optical flow to guide the warping process of noise.

- Inpainting examples and usefulness?
  Do the warped noise and self-guidance Equivariance work for generating "moving objects" in video inpainting? According to the formulation and the generation results, it seems it will favor generating "still objects" in the video inpainting case because it will easily meet the "warped" mechanism.

- Experiment Design.
  Experiments in Section 4.1 are not needed (or do not provide a useful argument) in the paper. Previous works have shown that warped noise is better than independent noise across the frames in a video. The only contribution here is that the fine-tuned version performs better than the un-fine-tuned version. The finding seems unrelated to the paper or provides quite limited information.
questions: questions: - A better way to illustrate how the proposed Equivariance Self-Guidance works.
  - A proper figure would be better instead of just formulations.

- See Point 2 in Weakness; can the proposed method handle cases where the object is moving or has different actions in the video inpainting case instead of being still?
limitations: limitations: For the limitations, the authors mention that in some situations, the work may fail. It would be helpful to also provide the corresponding failing examples, such as the second and third examples mentioned in Appendix F Limitations.
paper_summary: paper_summary: The paper follows a previous work on noise-warping methods for video inverse problem. The authors proposed instead of giving temporally consistent noise maps to the DM, using a continuous function space representation for DM directly can serve more complex spatial transformations and results in better temporal consistency in the generated videos. They demonstrated the proposed method on video inpainting and super-resolution.
strength_weakness: strength_weakness: The key statement in the paper is the importance of the equivariance of the DM to achieve temporal consistancy. The authors argued that the previous work "How I Warped Your Noise" performed badly when the DM is not equivariant. However, there are no explicit examples or solid proof on this statement. The proposed method for noise warping is more like offering another perspective than showing the failing reasons of the other work.

Furthermore, for super-resolution experiments, the authors did not provide comparison against "How I warped your noise". Even though they elaborated that the authors of "How I warped your noise" acknowledged to them that the work does not result in temporal consistency, I believe it is still better to include the comparison given how many super-resolution cases were presented in that work. If they don't provide the code, the authors may consider modify the one they have for the inpainting experiments and list out the comparison for more convincing results.
questions: questions: 1. It is not clear to me why the function space diffusion models need to be equivariant with respect to the underlying spatial transformations. I can imagine the necessity, but in your derivation, I don't see the proof that you stated you provide. Can you provide explicit examples or solid proof on this statement?
2. Why in fig 3 how i warped your noise seems to be in the second place in terms of error, but in table 2, the method seems not to outperform any other baselines?
limitations: limitations: The authors did not address any limitations
paper_summary: paper_summary: This paper addresses the challenge of temporally correlated inverse problems by employing an image diffusion model. The authors introduce a technique termed Warped Diffusion, which incorporates an equivariance self-guidance mechanism. This mechanism ensures that the generated frames maintain consistency when subjected to warping transformations. The paper demonstrates the application of this technique in the context of video inpainting and video super-resolution through a series of experiments.
strength_weakness: strength_weakness: 1. The authors report the noise warping speed at Line 357. We recommend that the authors also include the complete inference time required to process an entire video, which would provide a more comprehensive understanding of the method's efficiency.

2. We suggest that the authors provide more video results to further demonstrate the capabilities and limitations of the proposed method in various scenarios.

3. It is observed that the inpainted regions in the video inpainting examples remain static, such as the cat in the video not exhibiting motion. All apparent motion seems to be attributed to camera movements, with no optical flow present within the inpainted regions. We would appreciate an explanation for this phenomenon and its implications on the video inpainting process.
questions: questions: See the weaknesses.
limitations: limitations: The authors have adequately discussed the limitations and negative societal impact in the paper.
paper_summary: paper_summary: This paper aims to address video inverse problems using image diffusion models. By viewing videos as sequences of warping transformations between frames, the authors propose a novel approach called warped diffusion from a continuous function space perspective. Specifically, warped diffusion includes a Gaussian process-based noise-warping technique and a post-hoc equivariance self-guidance method to enhance temporal consistency. Experiments on video super-resolution and inpainting validate the effectiveness of warped diffusion. Moreover, warped diffusion first demonstrates superior performance with latent diffusion models such as SDXL.
strength_weakness: strength_weakness: 1. As illustrated in Table 1 and Figure 3, the proposed GP Noise Warping performs worse than "How I Warped Your Noise" in terms of warping error. The core performance improvement appears to come from the equivariance self-guidance.
2. As mentioned in Section G.3, the equivariance self-guidance is inefficient. Although the proposed GP Noise Warping is more efficient than "How I Warped Your Noise," the overall process can be more time-consuming.
questions: questions: 1. "How I Warped Your Noise" operates in a zero-shot manner using pretrained video editing diffusion models without requiring additional training. Is it possible for warped diffusion to also function in a zero-shot manner?
2. I am curious about the effectiveness of warped diffusion with just the equivariance self-guidance alone, without the GP noise warping.
3. What would the performance be like if the equivariance self-guidance technique were integrated into "How I Warped Your Noise"? Would it surpass the current performance of warped diffusion?
limitations: limitations: The limitations and societal impacts have already been discussed in the paper.

Comments: title: Thank you for acknowledging our rebuttal
Further Clarifications
Thank you for acknowledging our rebuttal
comment: Dear Reviewer,
thank you for reading our rebuttal; we are happy you appreciated the improved version. If there are no other major concerns, we would greatly appreciate it if you could increase your rating of our submission.

We understand your suggestion now and will incorporate it in the camera-ready version of our work to improve the presentation of our work.


Thank you again for your time and for helping us to strengthen our work!
Thanks for the clarification, and I'm happy to see the improved version. I might be confused by the wording in Section 4.1. I understand that performance improves after finetuning, which is already useful across all machine learning tasks and areas. This improvement is expected. Given the limited space, I would prefer combining Section 4.1 with other ablation studies and including more diverse experiments in the full paper.
We agree with the reviewer that we have not proven other approaches will not work. Indeed, let us consider the following argument. We'll work with $H$ a Hilbert space of real-valued functions over the domain $D = [0,1]^2$. Consider first the following definition of a video: A two-frame video is a pair of functions $f_0, f_1 \in H$ such that there exists a bounded, injective map $T: D \to \mathbb{R}$ with the property that $f_1(x) = f_0 \big ( T^{-1}(x) \big )$ for all $x \in T(D) \cap D$.

Now let $G: H \to H$ be a generative model for which there exists some $\xi_0 \in H$ such that $G(\xi_0) = f_0$. Then following holds: if $G$ is equivariant w.r.t. to $T$ then the pair $\big ( f_0, G(\xi_0 \circ T^{-1}) \big )$ is a two-frame video. This follows from the definition of equivariance as $G(\xi_0 \circ T^{-1}) = G(\xi) \circ T^{-1} = f_0 \circ T^{-1} $ on $T(D) \cap D$. Certainly, the other direction does not hold true. In particular, it does not follow that if $\big ( f_0, G(\xi_0 \circ T^{-1}) \big )$ is a two-frame video then $G$ is equivariant w.r.t. $T$. Therefore we agree with the reviewer that we have not proven that equivariance is necessary, however, we have proven that it is sufficient. We will explicitly point this out in the camera-ready version.

This lack of necessity stems from the fact that our definition of a video is weak and allows many possible pairs to be videos. One natural way of strengthening it is as follows: Let $\mathcal{T}$ denote the set of bounded, injective maps on $D$ then a two-frame video is a pair of functions $f_0,f_1 \in H$ such that the following problem admits a unique maximizer
\[\max_{T \in \mathcal{T}} \big \{ |T(D) \cap D| : f_1(x) = f_0 \big ( T^{-1}(x) \big ) \text{ for all } x \in T(D) \cap D \big \}.\]
In particular, we enforce that there is a unique deformation which keeps the maximum number of pixels in frame. From uniqueness, it now follows that: the pair $\big ( f_0, G(\xi_0 \circ T^{-1}) \big )$ is a two-frame video if and only if $G$ is equivariant w.r.t. to $T$.

We thank the reviewer for bringing this to our attention and are happy to discuss the mathematical modeling of videos in the camera-ready version further. 

If that's not what the Reviewer was looking for, please let us know and we will try our best to incorporate the Reviewer's feedback in our camera-ready.
Thank you for your response, especially for providing the comparisons for the super-resolution task and clarifying my misunderstanding on table 2. However, I am still not convinced about the "equivariant" statement. I understand your formulation and approach, but I don't agree other approaches "failure" automatically prove your statement.
We thank the Reviewer for acknowledging our rebuttal. We will make sure to include the discussions and experiments in the camera-ready version of our work.

We thank the Reviewer once again for their time and for helping us improve our work!
I appreciate the authors' efforts in addressing my concerns. After reviewing the other reviewers' comments and the authors' responses, I have decided to maintain my original score. 

I would like to see the authors include the discussions and experiments in the response to their final revision.