Infinite-Resolution Integral Noise Warping for Diffusion Models
Authors: Yitong Deng
Winnie Lin
Lingxiao Li
Dmitriy Smirnov
Ryan D Burgert
Ning Yu
Vincent Dedun
Mohammad H. Taghavi
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models; video generation; temporal consistency; multi-view consistency; noise warping; white Gaussian noise
Primary area: generative models

Abstract: Adapting pretrained image-based diffusion models to generate temporally consistent videos has become an impactful generative modeling research direction. Training-free noise-space manipulation has proven to be an effective technique, where the challenge is to preserve the Gaussian white noise distribution while adding in temporal consistency. Recently, Chang et al. (2024) formulated this problem using an integral noise representation with distribution-preserving guarantees, and proposed an upsampling-based algorithm to compute it. However, while their mathematical formulation is advantageous, the algorithm incurs a high computational cost. Through analyzing the limiting-case behavior of their algorithm as the upsampling resolution goes to infinity, we develop an alternative algorithm that, by gathering increments of multiple Brownian bridges, achieves their infinite-resolution accuracy while simultaneously reducing the computational cost by orders of magnitude. We prove and experimentally validate our theoretical claims, and demonstrate our method's effectiveness in real-world applications. We further show that our method can readily extend to the 3-dimensional space.

Published as a conference paper at ICLR 2025
INFINITE-RESOLUTION INTEGRAL NOISE WARPING FOR DIFFUSION MODELS
Yitong Deng!.? Winnie Lin', Lingxiio Li', Dmitriy Smirnov', Ryan Burgert 3,4, Ning Yu², Vincent Dedun', Mohammad H. Taghavi* NNetflx, Stanford University, NNetflx Eyeline Studios, <Stony Brook University
yitongd@stanford.ee
(winniel, lingxiaol, dimas, vdedun, mtaghavi (eghhawii@nettee fwweeefliixccom rburgert&cs st onybrook. edu
ning. www.eauliseneeeeeeae. com
ABSTRACT
Adapting pretrained image-based diffusion models to generate temporally con- sistent videos has become an impactful generative modeling research direction. Training-free noise-space manipulation has proven to be an effective technique, where the challenge is to preserve the Gaussian white noise distribution while adding in temporal consistency. Recently, Chang al. (024) formulated this problem using an integral noise representation with antees, and proposed upsampling- -aasee based algorithm to compute it. However, while their mathematical formulation advantageous, the algorithm incurs a high computational cost. Through analyzing the limiting-case behavior of their algo- rithm as the upsampling resolution goes to infinity, we develop an alternative algo- rithm that, by gathering increments of multiple Brownian bridges. achieves their anfinite-resolution accuracy while simultancously reducing the computational cost by orders of magnitude. We prove and experimentally validate our theoretical claims, and demonstrate our method's effectiveness in real-world applications. We further show that our method readily extends to the 3-dimensional space.
guar-
INTRODUCTION
The success of diffusion models in image generation and editing (Rombach et al. 2022 Nichol et al. 202 1
Ho et al.) 2020 Zhang et al. 2023a) has spurred significant interest in lifting these capacities
to the video domain Singer et al. 2022 Durrett 2019 Gupta et al. 2023 Blattmann et al. 2023 Ho al. 2022 Guo al. 2024). While training video diffusion models directly on spatiotemporal data is a natural idea, practical concerns such as limited availability of large-scale video data and high computational cost have motivated investigations into training-free alternatives. One such approach is to use pre-trained image models to directly generate video frames, and utilize techniques such as cross-frame attention, feature injection and hierarchical sampling promote temporal consistency across frames (Ceylan et al. 2023 Zhang et al. 2023b Khachatryan et al. 2023 Cong et al.) 2023).
Among these techniques, the controlled initialization of noise has been consistently shown to be an important one (Ceylan et al. 2023 Khachatryan et al. 20033 Cai et al. 20244.. However, most existing approaches for noise manipulation either compromise the noise Gaussianity (which intro- duces domain gap at inference time), or are restricted to simple manipulations such as filtering and blending which are insufficient for capturing complex temporal correlations. Recently, Chang et al. (2024) proposed method that both preserves Gaussian white noise distribution and captures temporal correlations via integral noise warping: each warped noise pixel integrates continuous noise field over polygonal deformed pixel region, which is computed by summing subpixels of an upsampled noise image. However, this method's 's theoretical soundness and effectiveness are ensued by its demanding computational cost in both memory and time, which not only incurs significant overhead at inference time but also limits its useability in novel applications (Kwak et al. 2024).
In this paper, we introduce a noise- warping algorithm that drastically cuts down the cost of Chang et al.) (2024) while fully retaining its virtues. Our key insight for achieving this is that, when adopting Fulerian perspective (as opposed the original Lagrangian one), the limiting-

---

Published as a conference paper at ICLR 2025
Undeformed
Deformed
Alternttive Procedure
S
Undefomed
Poal
Aggregate
Lagrangian
Eulerian
Noka Vector
Upxampled
Farttion
Viecoor
Uefformdd
Rogions
Figure I: When the image grid deforms, the Lagrangian view tracks deformed pixel region, while the Eulerian view tracks the undeformed pixel square as it gets partitioned into multiple regions. On the right, leverage the exchangeability of upsampled subpixels convert the Lagrangian gathering procedure into scattering noise subpixels to overlapped deformed pixel regions.
case algorithm of Chang et al. 2024 for computing warped noise pixel reduces to summing increments from multiple Brownian bridges (Durrett 2019, Section 8.4). In place of the costly upsampling procedure, sampling the increments of Brownian bridge can be done efficiently in an autoregressive manner (2). We build upon this to devise the anfinite-resolutionn integral noise warping algorithm (T) which directly resolves noise transport in the continuous space, when given an oracle that returns the overlapping area between a pixel square and a deformed pixel region (Section 2.2)
We propose two concrete ways compute this oracle, leading to grid-based and particle-based variant of our method. Similar to Chang et al. (2024), the grid-based variant (Algorithm ( com- putes the area by explicitly constructing per-pixel deformed polygons, and exactly equivalent to the existing approach (Chang et al., (024) with an infinite upsampling resolution, while running 8.0x to 19.7x faster and using 9.22x less memory" Inspired by hybrid Eulerian- Eulerian-Lagraagian fluid simulation (Brackbill et al. 1988), our novel particle-bused variant (Algorithm 3) computes area in a fuzzy manner, which not only offers a further 5.21x speed-up over our grid-based variant, but also agnostic non-injective maps. In real-world scenarios, the particle-based variant shows compromise in generation quality compared to the grid-hased one (see video results), while offering superior robustness, efficiency, simplicity, and extensibility higher dimensions.
In summary. we propose new noise- waiiinw method to facilitate video generation by lifting image diffusion models. Through analyzing the limiting case of the current state-of-the-art method (Chang et al. 2024) with an infinite upsampling resolution, we derive its continuous-space analogy. which fully retains its distributionnppeservinn and temporally coherent properties, while achieving orders- of- ma-maagni speed-up, warping 1024 x 1024 noise images in ~ 0.045s (grid variant) and ~ 0.0086s (particle variant) using laptop with Nvidia RTX 3070 Ti GPU.
2 METHODOL.OGY
In this section, we introduce our method follows:
We present an equivalent Eulerian interpretation (Figure [ of the method by Chang et al. (0244) which was originally described from Lagrangian viewpoint.
We show that the limiting algorithm of the Eulerian formulation as the upsampling level goes to infinity is equivalent to sampling increments of Brownian bridges.
We present our main algorithm (T) which, given a partition record that returns the over- lapping area between pixel square and deformed pixel region, samples increments of Brownian bridges and scatters the increments to form the warped noise image.
We propose two concrete algorithms for computing the overlap areas. The grid-based Algorithm extends Chang et al. (2024) to infinite resolution without the overhead of upsampling. The particle-based Algorithm 3 departs from grid-based discretization and uses particles instead, resulting in a simpler algorithm that is robust to degenerate maps.
2
'ince the official code of Chang al. 2024) is not available, performance is compared using our reimple- mentation in Taichi (Hu et 2019), which find he faster than reported in the original paper.
2

---

Published as a conference paper at ICLR 2025
Given a D D prior noise image Iw Dx and deformation map v : |0,1]² + [0, 1Ja. the norse-warping algorithm (Chang et al. 2024) computes the warped noise image Iw E DxD with upsampling level N E Σn1 as follows:
1. For Fori,j ,..., D, upsample noise pixel [Tw]k to N x N subimage [Tw]es ERNXN;
S N2 with Z NO 1) and - Zz.
N²
[wkks
N
The subimage for each pixel assembles into an ND x ND upsampled noise image Iw.
2. For j i..... D, the pixel square A., = 1 + 15 El is warped to deformed pixel region As, == (AL,), and the warped noise pixel [/w]k is set to be the sum of all subpixels in Tw covered by Ai. divided by VlAupl. where |A| denotes the Lebesgue measure of a Borel set A R??
We describe an alternative but equivalent procedure by making the following two observations, which are illustrated in Figure [
Gathering Noise - Scattering Noise. While the original procedure computes the warped noise im- age by gathering the upsampled noise subpixels in each deformed pixel region A, in Lagrangian fashion, we can instead use an alternative procedure by scattering the upsampled noise subpixels in each pixel square As, to overlapping deformed pixel regions. This new Eulerian procedure does not change the output, but yields new insights in conjunction with our second observation.
Scattering Noise + Counting Overlapping Subpixels. Observe that the N x N subpixels in [[w]. for every j, are correlated only through their sum S when conditioning on [[w]es (1, so they are exchangeable. Hence, when scattering these upsampled noise subpixels to deformed pixel regions, the order of scattering does not matter, and we only need count the number of subpixels covered by each deformed pixel region.
Alternative Eulerian Procedure. Putting both observations together, we now describe an alterna- tive procedure to Chang et al. (024) with unaltered output:
1. For each noise image pixel [Tw]aj. draw an upsampled subimage, represented
ID vector X E RN using 1b. Then, compute a prefix sum vector Hs via [Hr,lle =
ΣØ=1 Xx for k = I,..,,
2. Warp each pixel square and compute deformed pixel regions A.j as before.
3. For each Ar,j let M denote the number of deformed pixel regions that overlap with Ar,-- With index = ,,, we use 4,, my to denote the coordinates of the hthh overlap, whose pixel region is Arc,m, and pixel value [/wle.m, Form L E Z where La represents the number of upsampled subpixels covered by Aia.me- Then, compute a prefix sum [Ct,sle = Σ Tiq. For k 1,..., M, accrue (Haslio,sle [Hag]ic to J
4. Divide each warped noise pixel [Twlk, by v Vlrisl
Discussion. Compared to the original procedure by Chang et al. (2024), this alternative but equiva- lent algorithm highlights how the upsampled subpixels of [/w]s, are scattered to form the warped noise pixels. In particular, each warped noise pixel receives the sum of a segment of X, which computed by taking the difference of two entries of H.,-- Since Hy,, represents summations of weakly correlated and exchangeable subpixels, once conditioned on [[wls.j. instantiating every single subpixel, but instead model the sum of these weakly correlated subpixels?
avoid explicitly
The key insight of this paper is that when the upsampling resolution N - 00. the scaling limit of the prefix sum H,,; (with proper interpolation and time scaling to a continuous function) is precisely the Brownian bridge (Durreti 2019 Section 8.4) conditioned established, it is easy to progressively sample increments of the Brownian bridge, resulting in a clean and efficient noise- warping algorithm that bypasses the need for upsampling in Chang et al. (2024).
[ Once this connection is
Heee we assume that the noise image is square and has single channel only to simplify notation. In practice, the noise image can have arbitrary aspect ratio and number of ehaepes channels.
3

---

Published as a conference paper at ICLR 2025
Brownian Bridge B_()) with
MMM
Pixel Ay
10%
39%
15%
27%
9%
75%
8% 36%
Value Uw 0.45
9%
39%
10%
15%
Figure 2: Connection between Eulerian noise warping (scattering) and increments of a Brownian bridge for a fixed prior noise pixel [/w]..- The overlapping area with each warped pixel region becomes the "time increments" sampling the Brownian bridge these "times" and taking secutive differences yields the integral noise that is accrued to each warped noise pixel.
2.1 MINITE--ESSOLUTION NOISE SCATTERING
In this section, we first derive a scaling limit result to Brownian bridges. We then illustrate that the limiting version of the Fulerian procedure from the previous section matches precisely this scaling limit result. Lastly, we describe an autoregressive way to sample increments of a Brownian bridge that is lincar in runtime terms of the number of increments.
Theorem (Scaling limit Brownian bridge). Let { be sequence of i.i.d. random variables with finite variance that are normalized such that [[[Z]] = 0 and Var(Z,) = 1. For E IR, define
Sn
Sn
Z
X =
+ Vn
Z,
Consider the sequence of random continuous functions (H, (1) C(O, 1] defined as
Int] Ht((t) = X1n + (nt - [nt]) me]]+1,
Then the sequence {H.,} converges in distribution under the sup-norm metric C1O0, 1] to B.(()= = ((t) - (1) + tc, the Brownian bridge ending at c, where W(t) is standard Brownian motion.
Moreover, in distribution, we have (t(t) (W(t) ((1) = c), where (W W(1) c) the disintegrated measure (Pachl 1978 of W (L) ((1) =c
We prove Theorem Appendix A To connect the Eulerian procedure with the setup in Theorem Ct,; simplify the notation. By
let us fix pixel J.,. and let B Blwwk... H He,,, setting n = N2 and e = [/w]s,5, the sequence {Xum,} from the theorem has exactly the same law as the upsampled subpixels in [Twlig. Moreover, H.t = (t) when nt E Σz1- By taking N - xc, implying n - 0o, for any 1,...., IM [0, 1), we have the convergence in distribution of (H [Hin.).-... H [ $ (B(I)),.... (((())..,,B((())). Recall in the Fulerian procedure, only need to access the prefix sum at indices (Cr) where C₁ counts the number of upsampled subpixels covered by the first & overlaps. This suggests that if we choose
C N2
lim 00
A O A Aegromg
and use Btta) in place of HІ. then we just need to sample from B at times .i..... tas precisely the limiting algorithm of the Fulerian procedure. We illustrate this connection in Figure 2
4

---

Published as conference paper at ICLR 2025
Grid-based
Particle-based
Handling Degeneracy
Computo Polygon-Square Overtap Area
Compute Spatial Weighting Kamn!
Grid-based
Particle-based
Figure 3:
The grid-based variant (left) computes the overlapping areas by explicitly constructing the polygon for the deformed pixel region. The particle-based variant (middle) approximates these areas with weighting kernel. With degenerate maps (right), the fixed topology of the grid-based variant can lead to problems, while the connectivity-free, particle-based variant remains stable.
Autoregressive Sampling of Brownian Bridges. Since Brownian bridge is Markov process OOksndal 2013 Exercise 5,11), we can sample the vector (Be(f)),.... Be(tar) in an autoregres- sive fashion, each time sampling Bo(lx+1) conditioned on (t(l)):
(Be(k++1) Bx(+x) = =) = ( (1- 1x+1 Q
1x+1 - 4x (1++1 11)(1 18+1))
(2)
1 tx
Once the Brownian bridge at times tx is sampled, we just need to accrue the increments Be(tx) (l( to [/w]e the hthh overlapped warped noise pixel. This allows us to present Algo- rithm Compared to the discrete procedures described earlier, we no longer need upsampling. In addition, we exploited the autoregressive nature of Brownian bridges to bring down the time complexity to linear the number of overlapping warped pixel regions.
Algorithm
Integral Noise Warp
Input: prior noise image Iw DxD deformation map v [0, + [0,
Output: warped noise image Tw E RDD
Build partition record P from & (Section 2.2 Initialize Aij < for all i, = I..... ...., D. do 1, M < 0, Pu.ee
Ai, will contain the area of A
parallel for each v
for l = 1,..., M do
(a,i,)) [Puyle
is the overlapping between Aig and An,
Sample o' ~ (B. + B. = by with c == [ww].,
[wwks + [Twlas -9)
Aij + Aij +
q.t g
Normalize [[wks t Ai,j - [Twl, for all i,j D....,D
return Tw
Preservation of Gaussian White Noise. A central desideratum of noise warping is that the resulting warped noise image Iw needs to have pixels that are i.i.d. standard Gaussians when the prior noise image Iw is Gaussian white noise. This ensures that the warped noise is in-distribution for pre- trained diffusion model. Our algorithm automatically guarantees this preservation of Gaussianity, as long as the warping function d injective. To see this, the injectivity of yp implies that the warped
pixel regions are non-overlapping the square [0, 1]2 For each Ai.j since [T]),, d (0, W(1,, by the conditional interpretation of Brownian bridges ( when marginalizing out [Tw] li.j. the Brownian bridge Bpwk.. reduces to standard Brownian motion. Since the increments of the Brownian motion are independent Gaussians, the contribution to deformed pixel region is simply a zero-mean Gaussian with variance equal to the overlapping area. Therefore, each deformed pixel region will receive the sum of number of independent Gaussians whose variances to the of the region. The scaling by the inverse square root the area Algorithm T thus makes each warped noise pixel an i.i.d. standard Gaussian.
5

---

Published as a conference paper at ICLR 2025
2.2 BUILDING PARTITION RECORDS
To compute Algorithm 1 we need method to compute the partition record P which specifies how each pixel square is partitioned by multiple deformed pixel regions. In this section, we present one grid-based (Algorithm{2 ( and one particle-based (Algorithm{3 (3 method for building P. In particular, for each pixel square with indices (u. v). we compute Pu.e as a list of 3-tuples (a, i.j), where (i.j) identifies the overlapped deformed pixel region and represents the overlapping area.
Algorithm 2
Partition
Algorithm 3 Particle- based Partition
Input: Deformation map 1/ Output: Partition record P I: parallel each i. do
Input: Deformation map t
Output: Partition record P
1: parallel for each x.j do
2: A <- DiscretizeSquare(A,
2: (z. <- 56 ++006 240.5)
D D
3: S S+~(A")
3: Oa.u, Qu.1, Q1,0, 01.1 + Bilinnarweight((X
4: u " N0 , AABB(S)
4: for s, 0, do
for E [u =ut
5:
u. u.0+ [ +s |, +
for U E [U v" ] do
6:
Pe.r < Pa. + (0,,t,
7:
a + AregoonAreaCCli((5555 v))
7: parallel for each u, e do
8:
Pu, + Pa,. + (a,
8: Normalize total area of Pu,e to D-2
9:
return P
9:
return P
As illustrated in Figure 3 our grid-based method (left) follows Chang et al. (2024) by modeling each deformed pixel region as an octagon and computes overlapping areas by clipping it against undeformed pixel squares; our particle-based method (middle) borrows from the grid-to-particle techniques fluid particle-in-cell methods (Brackbill et al., 1988), where we treat each deformed pixel region particle and each undeformed pixel square grid cell. Each particle requests area from nearby cells based on distance; upon receiving requests, each cell normalizes the requests to ensure partition-of-unity. and distributes its area contacting particles.
Discussion. Conceptually, our grid and particle-based methods correspond to two different inter- pretations of da when provided discrete samples (e.g., optical flow image). The grid- grid-based method implicitly reconstructs the continuous vi field by linear interpolation, whereas the particle- based method assumes is only known point-wise. The implication is that when is smooth, linear interpolation works well and the grid-based method will yield a higher-quality warp as seen in Fig- ure B.3 But when e is non-smooth, which is commonly the case in real world, linear interpolation can lead to degenerate polygons as illustrated on the right of Figure 3 The spurious overlaps be- tween the degenerate polygons will lead to spatial correlation in the warped noise image. Although both both|Chang et al. (2024) and our grid-based method implement fail- fail-safe to avoid noise sharing and maintain spatial independence in practice, they suffer from the intrinsic ambiguity caused by these overlaps. On the other hand, the particle-based method circumvents such overlaps to begin with.
In addition, we highlight the simplicity and parallelizability of the particle-based method, as boils down the computation of P to evaluating one bilinear kernel per pixel. Leveraging this fact, we can very conveniently and efficiently extend our noise warping algorithm to higher spatial dimensions by replacing the bilinear kernel to its dimensiodma counterparts, as shown in Figure g
RESULTS
We verify our theoretical claims by showing that both variants of our method preserve Gaussian white noise distribution, and that Chang et al. (2024) (HIWYN) converges to our grid-based variant as N increases. We analyze the behaviors of our grid and particle-base variants under diffeomor- phic and non-diffeomorphic deformations. We then apply our method to video generation tasks and
benchmark against existing methods (Ge et al. 2023 Chen et al. [023 Chang et al. 2024). Finally, we extend our method to warping volumetric noise and demonstrate use case in 3D graphics.
Gaussian White Noise Preservation. In Figure we iteratively warp noise image by the same deformation map for 50 timesteps. We gauge the output noise's 's resemblance to Gaussian white noise through 1) measuring normality using one-sample
Saamogorov-mmimoon (K-S) test, and 2) detecting
In our case, clamp the input fx+1 to hefore sampling by 2 Intuitively, this means that when an undeformed pixel has assigned its entire pixel region deformed pixels, later noise requests will be neglected.

---

Published as a conference paper at ICLR 2025
Ours, grid-based
Ours, bared
HIWYN N
Bilinear
Bicubic
Distribution Preservation Metrics
Method
Ours, grid-based
Ours, basee
IIWYN, 8
Bilineur
Bicubic
Nearest Neighbor
Moran's /
1103c-4/0.800
-1.995c-3 0.445
3.215e-3/0.243
/0
0,983 /0
2.974c-2/6 6,13c-27
K-S Test
3.410e:3/ 0.430
3.023e-3 0.586
3.274e-3/0.482
0.366
0.422
9.806e-3 14066-3/6.681c-000
Preservation of Gaussian white noise achieved by different warping methods. We report scores and p-values for both Moran's / (spatial correlation) and K-$ test (normality). Under these metrics, the results from our method (both variants) and HIWYN are indistinguishabl from white Gaussian noise, while generic, interpolation-butt warping methods lead to corrupted noise.
Figure 4:
Map
0.234
Insant
Varying
mean
W
max
mean W
max
0:3-
2.072e-1
7.253e11
4.320c-3
1.391e-2
5,394e-2
2.962e-1
3,236e-3
8,325e-3
52
8
1.881e-2
8,310c-2
3.616e-3
8.134e33
41
16
6.792e-3
2.361e-2
3.387e-3
8,228e-3
106
R-2
N-8 N-16 N-32 N-64 12-128
25256
Figure 5: Convergence of HIWYN to method as N increases. Top left: experimental setup with prior noise and deformation map. Top middle: 2-Wasserstein distance W between the output of HIWYN and ours. Top right: statistics table. Bottom: WN difference image between the output of HIWYN and ours as N increases. W becomes statistically insignificant for N 2 64.
spatial correlation using Moran's T. Our results show that both HIWYN and our method generate warped noise images that are indistinguishable from Gaussian white noise under these metrics, while interpolation based basclines deviate significantly from the desired distribution.
Convergence of C'Cann et al. (024).. We validate that method is the limiting Starting with an x prior noise image and flow map (Figure top left), we run our method along with HIWYN for N E {2, 8, ... 256} for 100,000 independent runs estimate the distribution of the warped noise image. For each upsampling resolution N, we compute the 2- 2-Wasserstein distance W. between the output of HIWYN and that of method. The results in Figure 5 demonstrate the convergence of HIWYN to our method as V increase, and reveal that N = (recommended by Chang et al. 2024 is not yet in the converged phase to yield negligible W
of HIWYN.
Performance Comparison. For our methods and HIWYN with upsampling levels N E {2, 4, 8), perform 100 independent runs
1024 x 1024 image. We report the kernel time with CPU and GPU backends (Figure 6) as well as the memory usage. The runtime and memory usage of our methods are largely comparable with those of HIWYN with N = 2. Compared to HIW YN with N W-8 8, both variants of our method offer order-of- ofatr--f-mmanitude improvements in runtime and mem- ory usage. Specifically, our grid-based method achieves infinite upsampling resolution while being 19.7x faster CPU and 8.0x faster GPU, using 9.22x less memory, and our particle-based method, albeit not strictly equivalent to HIWYN at = co, achieves a 41.7x speedup on GPU. In the following sections, show that the particle-based based version consistently achieves comparable quality to the grid-based version in real-world scenarios.
Comparison between Grid-Based and Particle-Basse Variants. In Figure B.3 compare both variants when the deformation map is diffeomorphic under different levels of distortion. Visually, the difference between the two variants is negligible frame 25 and becomes noticeable in frame

---

Published as a conference paper at ICLR 2025
Performance Comparison
Comparison of Time and Memory Costs
Method
Time (CPU)
Time (GPU)
Memory
Ours, and hased
2 (HIWYN)
19.30s
2.597s
293.7MB
N 4 (HIWYN)
75.33s
9.247s
746.6MB
HWWWN, N
N 8 (HIWYN)
398.3s
35.91s
2147MB
HWVZK, N
N (ours, grid)
20.16s
4.491s
232.9MB
N 0o (ours, particle)
15.68s
0.862s
320,9MB
(CFU)
(GPU)
Memory
Figure 6: Runtime and memory usage our method versus HIWYN with N 2, 4, We compare total allocated memory and kernel time CPU/GPU. The computation is done a laptop with Intel 17- 12700H and Nvidia RTX 3070 Ti.
Video
Optical Flow
Grid-based
Particle-based
Area Occupation (Grid-based)
Area Occupation (Particle-based)
Figure 7:
Comparison of grid-based versus particle-based variants under non-diffeomorphic optical flow. Pixels with area contention are colored in orange. Further results are given in Figure B.5
100. We measure this difference by comparing the deformed regions for each pixel in terms of IoU and weighted Chamfer distance. We additionally compare the particle-based result with that of an identity-map baseline (right column Figure B.3). which shows that the gap between the two variants remains small even under large distortion. In Figure 7 we stress test both variants under non-diffeomorphic maps obtained using optical flow (Teed & Deng 2020) on a real-world video (Brox& & Malik (011). In panels 3 and 4. we see that the real-world flow map induces inverted meshes for the grid-bases variant and clustered particles for the particle-based variant. While clus- tered partieles
always assigned disjoint regions due to the continuous nature of our algorithm, mesh inversions cause area contention issues (non-injectivity) due to overlaps. In panels and 6, we mark the grid cells with area contention orange, which occurs in the grid-based version but not in the particle-based version.
Video Super-resolution with ISS.. We integrate our method with ISSB (Liu et al. 2023 and adapt its pre-trained image 4x super-resolution model (bicubic) to perform video super-resolution. We show our results in Figures 2 and B.I. and refer to our supplementary video for better visualization of these results. Since SB is an image-to-image bridge model, it well preserves the low-frequency structures of the input images regardless of noise scheme. But as seen our video, without noise warping, the results either show strong flickering in the figh-freeuency details (random noise) or sticking artifacts (fixed noise). Noise warping allows high-frequency details to transport with optical flow, making the result significantly more consistent. We also validate that both our variants yield visual quality on par with HIWYN across all tested scenarios while being much more efficient.
Conditional Video Generation with SDEdit. We apply our method to conditional video genera- tion by adapting SDEdit (Meng et al. 202I), a conditional image generation method, to producing consistent video frames. We set SDEdit's s parameter t to 0.4, and apply Perturbed-Attentee ance (Ahn et al. 2024) with strength 3.0. Our two inputs are a conditioning video (generated by applying median filter to
Guid-
videos similar (Chen al. 2023)) and optical flow field (Teed & Deng 2020). Without noise manipulation, if we run SDEdit independently frame-by-frame (Figure B.6 random), the high-frequency details display significant flickering. By warping the noise using the optical flow, the temporal consistency is much improved, and we observe that our method (both variants) and HIWYN yield comparable visual qualities. Full experimental results with com- parisons to Control-A-Vice are provided in Figures B.6 and B.7 with quantitative statistics reported Table [ Further results that additionally integrate cross-frame attention (Ceylan et al. (023) (anchored every 3 frames) are shown in Figure 10 and B.2 We refer to our supplementary video for better visualization.
(Chen et al. 2023), PYoCo (Ge et al. 2023) and additional baselines
8

---

Published as a conference paper at ICLR 2025
Input
Output
Figure 9: Video 4x super-resolution by integrating our method (particle) with SB Top row shows the low resolution input video; bottom row shows the output video. Additional results are shown in Figure B.I
We refer to our supplementary video for better visualization these results.
Input
Output
Figure 10: Conditional video generation results by integrating our method (particle) with SDEdit and cross-frame attention (Ceylan et al. (023). Top shows the input video prompt in stroke painted style which converted into a video of photorealistic style (bottom). Additional results are shown in Figure B.2 We refer
supplementary video for better visualization of these results.
3D Noise Warp. We extend our particle-based algorithm to 3D by re- placing the bilinear kernel with the trilinear kernel in Algorithm 3 and apply it to GaussianCube (Zhang et al.] 2024), which denoises dense 3D noise grid to reconstruct 3D Gaussians. We adapt it to perform condi- tional generation a la SDEdit. Starting with a 3D pickup truck generated unconditionally, we condition the model generate vehicles with smaller and larger cabins by deforming the truck with a horizontal shear velocity field. We compare the results obtained by using random noise to those ing noise warped with our particle-based method. We show the results Figure B.4 and refer our supplementary video for better visualization.
Figure 8: 3D noise warped by our method.
RELATED WORKS
Noise in Diffusion Models. Diffusion models generate images from input noise, and noise can thus be considered the counterpart to the latent codes utilized in GAN models. As such, the outputs of diffusion models have dependencies and correlations to the initial input noise, making noise useful handle control temporal consistency (Khachatryan et al. (023). In addition Chang et al. (2024) which this work was inspired by and improves upon, there are various other temporal noise manipulation techniques that not preserve Gaussian noise distribution- some methods (Ma) et al. (2024); |Ren et al. (2024)) blend high frequency Gaussian noise with low frequency motion, while others (Mokady al. (2022); Wallace et al. (2022)) rely on approximating the inversion of noise from temporally coherent image sequences. Pandey et al. (2024) goes one step further and manipulates inverted noise in 3D space. These approaches are flexible but degrade the output of the diffusion model due to the domain gap between inference time noise and training time noise, and as such, have occasionally been accompanied by mitigation strategies such as anisotropic diffusion Yu et al. 20244)) Noise manipulation is also not limited to the generation and stylization of videos,

---

Published as a conference paper at ICLR 2025
but has various applications in image editing (Hou al. (2024); Pandey et (024)) and 3D mesh texturing (Richardson et al. ) as well.
Noise in Computer Graphics. While our noise warping work draws main inspiration from simula- tion techniques, spatial noise manipulation has been extensively studied in the graphics community through applications in animation and rendering. Works like (Kass & Pesare 2011 Burley et al, 2024] present 2D noise manipulation techniques that add stylized organic hand drawn look to computer-generated animation via dynamic noise textures (Perlin) 1985). In order to make sure the stylization is temporally consistent and visually pleasing, noise textures are deformed in a way that makes them consistent with the underlying animation, but little emphasis is given to the preserva- tion/rigor of the noise distribution. On the other hand, properties of 2D spatial noise have been extensively and rigorously studied in rasterization and raytracing literature (Cook) 1986; Lagae & Dutré 2008), originating from the idea of using dithering to reduce banding and quantization arti- facts in image signal processing (Roberts) 1962). In particular, the lack of low frequency details and clumping in blue noise opposed to white Gaussian noise has made it the choice of foundational antialiasing methods such as Poisson disc sampling (McCool & Fiume 1992p, and recent progress made this line of antialiasing research has close ties with our methodology. For example, WWolfe itt al. (2022) look at accelerating rendering tasks by extending spatial blue noise to the temporal domain, while Huang al.) (2024) show promising results in supplementing white noise with blue noise during diffusion model training.
5 DISCUSSION AND CONCLUSION
We presented infinite- integral noise warping, a novel algorithm for computing temporally coherent, distribution- preeriovinn noise transport to guide diffusion models to produce consistent re- sults. By deriving continuous-ppace analogy to the existing upsampling-bass ct al. 2024), our method not only further improves the accuracy by effectively raising the upsam- pling resolution to infinity, but also drastically reduces the computational cost.
strategy (Chang
Usability of Noise Warping We highlight that the noise warping problem that we address is a recurring subtask in generative modeling, and our method is hence a general-purpose tool that can be integrated in a variety of ways that extend well beyond the ones we showcase in the paper. First, noise warping, which excels at controlling high-frequency details, is orthogonal and thus com- binable with feature-level, prucctre-preservvee
techniques (e.g. [Cylaan et al. (2023); Cong et al.) (023 ) to achieve consistency across the frequency spectrum. Our drastic sast-n makes noise warping an affordable and harm-free free add-on to all such existing and future techniques. In addition, the concurrent work by Daras et al. (2024) shows that noise warping can be combined with equiv- ariance guidance to gain further consistency and integrate with latent diffusion models like SDXL (Podell al. 2023). Beyond video generation, Kwak al. (2024) showcases the usefulness of noise warping in 3D generation by combining with score distillation sampling (SDS). The advanced noise warping algorithm that we propose presents itself a desirable candidate across these diverse tasks.
Significance of Our Speed-up We argue that the drastic speed-up our method offers has profound practical significance. While the standard denoising diffusion setting requires only a single noise warp operation per image, there exist many use cases that require noise warping to be computed more intensively, which renders our speed-up critical. For example, the combination with bridge models (e.g. 1SBB) requires one noise warp per iteration. With its reported > 0.6s time cost per warp, preparing the noise using HIWYN would cost 4x the time actually run the image generation model, increasing the total inference time from ~ 24 minutes to ~ 2 hours. In comparison, our method (particle) prepares the noise in 40.6s (wall time), effectively making the overhead negligible. Similarly, combining noise warping with SDS also requires one noise warp per iteration, which makes HIWYN computationally intractable (Kwak et al., 2024) and our improvements called for. Our speed-up hence makes integral noise warping deployable in much broader class of problems.
We identify a few directions for future work. First, our particle-based variant currently does not cap- ture area contraction or expansion which may be addressed in the future with Voronoi partitioning. Secondly, our method can be readily combined with other map types such as UV maps to facilitate multi-view consistency. Furthermore, since method relies the consistency and quality of the deformation map, it can benefit from future advancements in flow extraction techniques. Finally, the effectiveness of warped volumetric noise in 3D generation and editing remains to be studied.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank Lukas Lepicovsky, Ioan Boieriu, David Michielsen, Mohsen Mousavi, and Perry Kain from Eyeline Studios, for providing data that kickstarted this project and for assisting in shaping our research with practical future use cases. We also thank Austin Slakey for sharing his insights on training diffusion models, and Tianyi Xie for filming his cat for our testing.
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jacwon Min, Wooscok Jang, Jungwoo Kim, SconHwa Kim, Hyun Hee Park. Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. arXiv preprint 2rXiv:2403.17377 2024.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Jeremiah U Brackbill, Douglas B Kothe, and Hans M Ruppel. Flip: a low-dissipation, particle-in- cell method for fluid flow. Computer Physics Communications, 48(1):25-335 1988.
T. Brox and J. Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 33(3):500-513, cat
2011. URL http:://pmm informat. k. uni free Brolla
Brent Burley, Brian Green, and Daniel Teece. Dynamic screen space textures for coherent styliza- tion. In ACM SIGGRAPH 2024 Talks, SIGGRAPH '24, New York. NY, USA, 2024. Associa- tion for Computing Machinery. ISBN 9798400705151. doi: 10.1445/644133.3664421. URL.
htttps://doi.om wrg/rrrgg00.. /111/1141//331133 3664321
Shengqu Cai, Duygu Ceylan, Matheus Gadelha, Chun-Hao Paul Huang, Tuanfeng Yang Wang, and Gordon Wetzstein. Generative rendering: Controllable 4d-guided video generation with 2d diffusion models. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 761 1-6620,,, 2024.
Duygu Ceylan, Chun-Har P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23217, 2023.
Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C Azevedo. How warped your noise: correlate noise prior for diffusion models. The Twelfth International Conference on Learning Representations, 2024.
Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305. 13840, 2023.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Mannell Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing. arXiv preprint urXiv:2310.05922, 2023.
Robert L Cook. Stochastic sampling computer graphics. ACM Transactions on Graphics (TOG), 5(1):51-72, 1986.
Giannis Daras, Weili Nie, Karsten Kreis, Alex Dimakis, Morteza Mardani, Nikola Borislavov Ko- vachki, and Arash Vahdat. Warped diffusion: Solving video inverse problems with image diffu- sion models. arXiv preprint arXiv:2410.16152, 2024.
Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22930-22941, 2023.

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. International Conference on Learning Representations, 2024.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.0666 066622. 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02.0 10.02303 2022.
Xingzhong Hou, Boxiao Liu, Yi Zhang, Jihao Liu, Yu Liu, and Haihang You. Easydrag: Efficient point-based manipulation on diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8404-8413, June 2024.
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: a language for perehpeerfrmaeee computation on spatially sparse data structures. ACM Transac- tions on Graphics (TOG), 38(6):1-1,, 2019.
Xingchang Huang, Corentin Salaun, Cristina Vasconcelos, Christian Theobalt, Cengiz Oztireli, and Gurprit Singh. Blue noise for diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Michael Kass and Davide Pesare. Coherent noise for non nonpphotorcalistic rendering. ACM Transac- lions on Graphics (TOG), 30(4):16,, 2011.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero, Text-to-image diffusion models zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954-15964, 2023.
Min-Seop Kwak, Donghoon Ahn, Ines Hyeonsu Kim, Jin-wha Kim, and Seungryong Kim. Geometry-aware score distillation via 3d consistent noising and gradient consistency modeling.
arXiv preprint arXiv:2406. 2024.
Ares Lagae and Philip Dutré. comparison of methods for generating poisson disk distributions. In Computer Graphics Forum, volume 27, pp. 114-129. Wiley Online Library, 2008.
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar. Fbb Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.0587., 2023.
Xin Ma. Yaohui Wang, Gengyu Jia, Xinyuan Chen, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Cin- emo: Consistent and controllable image animation with motion diffusion models. arXiv preprint arXiv:2407.15642, 2024.
Michael McCool and Eugene Fiume. Hierarchical poisson disk sampling distributions. In Graphics interface, volume 92, pp. 145, 1992.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2/08.01003, 2021.
Ron Mokady, Amir Hertz, Kfr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:221 2022.
Peter Mörters and Yuval Peres. Brownian motion, volume 30. Cambridge University Press, 2010.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with lext-guided diffusion models. arXiv preprint arXiv:21 2021.
12

---

Published as a conference paper at ICLR 2025
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Jan K Pachl. Disintegration and compact measures. Mathematica Scandinavica, pp. 157-168, 1978.
Karran Pandey, Paul Guerrero, Matheus Gadelha, Yannick Hold-Geoffroy Karan Singh. and Niloy J Mitra. Diffusion handles enabling 3d edits for diffusion models by lifting activations to 3d. Pro- ceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp- 7695- 7704, 2024.
Ken Perlin. An image synthesizer. ACM Siggraph Computer Graphics, 19(3):287-22) 1985.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01195, 2023.
Weiming Ren, Harry Yang. Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint 24X2:22402.04324,, 2024.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- guided texturing of 3d shapes. In ACM SIGGRAPH 2023, 2023.
L.. Roberts. Picture coding using pseudo-random noise. IRE Transactions Information Theory, (2):145-154, 1962. doi: 10.1109/TT.119221057702
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. vvieeo Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14770 2022.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16. pp. 402-419. Springer, 2020.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled trans- formations, 2022. URL http:::/aaxxv..oo/////oa///
12446
Alan Wolfe, Nathan Morrical, Tomas Akenine- and Ravi Ramamoorthi. Spatiotemporal Blue Noise Masks. In Abhijeet Ghosh and Li-Yi Wei (eds.), Eurographics Symposium on Rendering. The Eurographics Association, 2022. ISBN 378-3-03868-187-8 doi: 10.2312/st 161.
Xi Yu, Xiang Gu, Haozhi Liu, and Jian Sun. Constructing non-isotropic gaussian diffusion model using isotropic gaussian diffusion model for image editing. Advances in Neural Information Processing Systems, 36, 2024.
Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, and Baining Guo. Gaussiancube: Structuring gaussian splatting using optimal transport for 3d generative modeling. arXiv preprini 2.0i::2003.195555 2024.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image the IEEE/CVF International Conference on Computer Vision,
diffusion models. In Proceedings the pp. 3836-3847, 2023a.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training controllable text-to-video generation. arXiv preprint arXiv:2305.13077 2023b.
13

---

Published as a conference paper at ICLR 2025
A PROOF OF THEOREM
Proof. By unrolling the definitions, for & E [0, 1), we have
Lie]
1
(t) = (t) - (1) + tc, (t) = vn
Σ
Zi + (nt - [[nt] 2 (mtt]++
By Mörters & Peres
(2010 Theorem 5.22), {S converges in distribution to W (t) under the sup-norm metric of C Ij. To lift this convergence to the sequence {Hn (nZz1 observe that the function C 1] + C 1] defined by
g(x(t)) = x(t) - tx(1) + to
continuous under the sup-norm metric. {fr [falnezgi. C(O,, 1]. Then
To verify this, suppose lim In f for
(lo(fa) - -(f)ll sup (t) (f((1) + tc) (f(t) tf(11 tc)l =e(0.1]
Ifff - fl + (1.a((1) - f(1) s 2 - -fllc + O.
Hence, by the continuous mapping theorem,
g(S))) Hn - B(t) ([t)-tB(1)) tc.
To show
((t) -(t)-WW()) t to= (W W(1) c),
first of all, the conditioning (W(+) ((1) =) is interpreted as the limit of (W(t) [W(1) - c/<
e) as - O. Denote (t) W(t) (1.. so that W(t) Y() + (1). Since
(Y (L), =
(1)) = (C((,, ((1)- (W))))(1)))
and that (1)) are jointly Gaussian, they are independent. Therefore,
lim (W(() [ <
e) <) -00 (() + (W()) [W(1) c <)
(t) + -0
lim (4) [W d <)
((t) (1) tc.
B ADDITIONAL RESULTS
In this section we include additional visual and numerical results. Figure B.1 shows additional video 4x super-resolution results with IB in addition to those in Figure 9 Figure B.2 additional video generation results with SDEdit and cross-frame attention, which mirrors the setup of Fig- ure TO Figure B.6 and Figure B.7 show video generation results with SDEdit only (without cross- frame attention) isolate the influence of noise initialization, and we compare our method (both variants) with HIWYN (Chang et al. 2024). Control-A- Control-A-Video (Chen et al., 2023) PYoCo (Ge et al., 2023), as well as baselines with random, fixed, and interpolated noise using bilinear and nearest interpolating schemes. The statistics for both the church and cat scenes are reported in Table Fig- ure B.3 demonstrates the differences between both our variants computing warped pixel regions in the diffeomorphic case. Figure B.5 shows the differences in the non-diffeomorphic case, where we use additional examples to demonstrate the area contention issue caused by degenerate meshes that applies similarly to grid-based variant and (Chang et al. 2024), and highlight the robustness of our particle-base based variant. Figure B.4 shows additional results for applying our particle-based based, volumetric noise warping 3D tasks with GaussianCube (Zhang al. 2024).
14

---

Published as a conference paper at ICLR 2025
Video Generation Quality (Church)
Metric
Ours (G)
Ours (P)
HIWYN
PYoCo
CaV
Random
Fixed
Bilincar
Nearest
Consistency
6.955e-2
7.057e-2
6,953e-2
9. 9.800e-2
1.108--1
1.150e-1
9.273e-2
6.906e-2
8,985e-2
Realism 4
7.251e-2
7.130e-2
7.197e-2
7.535e-2
7.430e-2
6,336c-2
5.893e-2
2.102e-1 1
8,194e-2
Faithfuiness +
1.11le-2
1.166e-2
1.171e-2
1.184e-2
1.555e-2
1.344e-2
1.030e-2
1.883e-2
3.090c-2
Vidco
Generation Quality (Cat)
Metric
Ours (G)
Ours (P)
HIWYN
PYoCo
CaV
Random
Fixed
Bilincar
Nearest
Consistency
+
4.016e-2
4.070c-2
3,810c-2
5.995e-2
4.507e-2
1.093c-1
4.537e-2
2.613e-2
7.974e-2
Realism
2.235e-1
2.126e-1
2.108e-1
1.67e-1
1.931e-1
1.56le-1
1,982e-1
3,271e-1
3,195e-1
Faithfulness
7.841e-3
7.556e-3
7.305e-3
9. 9.2900-3
7.427e-3
9.637e-3
8,558e-3
1.162e-2
1.128e-1
Table I: We show the quantitative statistics for conditional video generation using SDEdit without cross-frame attention. The consistency is measured using Warp MSE (Chang et al. 2024). and the realism and faithfulness are measured as Meng et al. [20IT).
Input
Output
Input
Output
Input
Output
Figure B.1: Additional results generated by performing 4x video super-resolution with PSB. For each scene, the upper row represents the low resolution input video, and the lower row represents the high resolution output video generated with our particle variant. We refer to our supplementary video for better visualization of these results.
15

---

Published as a conference paper at ICLR 2025
Input
Output
Figure B.2: Additional conditional video generation results by integrating method (particle) with
SDEdit and cross-frame attention (Ceylan et al. 2023 on a cat scene.
Time
Frama 25
3.0
6.0
Frame 50
Iuselling
20
40
60
00
300
busclin particee
Frame 100
Flow Map
Grid-based
Particle-based
loU
Chamfer
Figure B.3: Comparison between the grid-based and particle-based variants for building partition records when the deformation map is diffeomorphic. The first column shows the deformation map different frames. The second and third columns visualize warped pixel regions for both methods. The fourth and fifth columns show the IoU (larger is better) and Chamfer distance (smaller is bet- ter) between the warped pixel regions of both variants. We plot the distance between particle and grid variants alongside baseline, which is the distance between identity map and the grid variant, to verify that the particle-based version remains close to the grid-based version even under large distortion.
Volume View
Slice View
3DGS View
3D Warped
Defom lift
Dafom
Random Nolse
Figure B.4: Extension of our particle-basee method to warping 3D, volumetric noise. We show volume renders of the noise the top left, slice views the top middle, and 3D Gaussians used the GaussianCube representation on the top right. The bottom row shows the respective results obtained by running 3D generation with warped and random 3D noise.
16

---

Published as a conference paper at ICLR 2025
Video
Optical Flow
Grid-based
Particle-based
Area Occupation
Area Occupation (Grid-based) (Particle-based)
Figure B.5: Comparison of grid-based and particle-based variants under non-diffeomorphic defor-
mation maps typically found in real-world applications. The orange pixels are the invalid pixels where area contention occurs. Flow maps are downsampled 10x for better visualization. Image se- quence comes from Brox & Malik 2011) while optical flow is computed via Teed & Deng 2020).
17

---

Published as a conference paper at ICLR 2025
Input
Ours (grid)
Ours (particle)
PYoCo
CAV
Fixed
Random
Nearest
Bilinear
Figure B.6: The full results generated by all the compared methods on the church scene. The
interpolation basclines yield noticeably corrupted results, and others yield similar quality the image level. The difference lies in how the details are preserved across frames. We refer to our supplementary video for better visualization of these results.
18

---

Published as a conference paper at ICLR 2025
Inpul
Oura (grid)
Ours (particle)
HIWYN
PYoCo
CAV
Random
Nearest
Bilinear
Figure B.7: The full results generated by all the compared methods on the cat scene. We observe that HIWYN and our method (both variants) yield similar results. While both our variants much faster and more memory-efficiien than HIWYN, this makes a particularly strong case for our particle-based variant considering its significantly improved simplicity and efficiency. We refer to our supplementary video for better visualization of these results.
19


Reviews:
[{"review_id": "Aj4wdgS6y6", "reviewer": "Reviewer_jdjC", "summary": "The topic of the paper is to develop computationally efficient algorithms to use image diffusion models temporally, i.e. for generating consistent videos or solving video inverse problems. The authors innovate on the noise-warping mechanism and they propose a computationally efficient method to transform the noise between frames that preserves the Gaussian prior distribution. The authors experimentally validate their method for video super-resolution and 3-D problems.", "questions": "See also weaknesses above.\n\n1) The experimental results in the website of the How I Warped Your Noise paper (https://warpyournoise.github.io/), seem of much higher consistency compared to the ones provided in the zip file by the authors. Could the authors comment on this? Were there any reproducibility issues?\n\n2) This [paper](https://arxiv.org/pdf/2410.16152) claims that \"white noise is not compatible with the idea of using a generative model to interpolate deformed functions\". Could the authors comment on this and the preservation of the Gaussian prior under different transformations?\n\n3) Could the authors investigate if having an equivariant generator improves the experimental results?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1) The paper is very well written and the ideas are presented in a natural way.\n\n2) The method is significantly faster compared to the baseline.\n\n3) The fuzzy variant of the method is very clever: significant computation time is saved and the method is more robust as evaluated in real-world applications.\n\n4) The ideas presented are sound and the Euclidean View offers an alternative way to think about the problem of noise warping that I enjoyed reading about.", "weaknesses": "1) The biggest weakness of this paper is that the experimental results are very weak. I downloaded the supplementary material and looked at the videos; the quality is unsatisfactory. While the proposed method offers more consistency than the existing baselines, it is still very inconsistent. The extent of inconsistency is so high, that I can't think of any applications that would leverage this method. \n\n2) At the same time, I am not very confident that the considered approach is meaningful. It is true that at the moment there are many powerful image diffusion models and not so many video diffusion models. However, the current level of quality of solving inverse problems with image diffusion models appears to be very low, judging from the results of this paper. I am not sure if this is a fundamental limitation of the approach or whether it will be resolved in the future. But also, as time goes by, more and more powerful video diffusion models are becoming available that can be used as priors for solving temporal inverse problems.\n\n3) I believe that one of the reasons the experimental results are not that strong is the equivariance of the network itself. Even if the input noise transformations are properly handled, maybe the network itself does not provide equivariant predictions which could cause the creation of flickering and inconsistencies. This is argued in this work: https://arxiv.org/pdf/2410.16152. \n\n4) I also believe that the paper would benefit from more experiments on other inverse problems. Since a pre-trained prior is available, one could experiment with other types of problems such as inpainting or even denoising as the image follows simple geometrical transformations (zoom-in/out, shifts, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DzbjfY1iPh", "reviewer": "Reviewer_p4MX", "summary": "The paper proposed a different formulation for the \"integral noise warping\" approach of Chang et al. (2024). The authors prove the relation of this formulation (in infinite resolution) to Brownian bridges, and use it to derive novel implementations for the integral noise warping that do not require upsampling, and thus significantly reduce the computational cost of the method proposed by Chang et al. (2024).", "questions": "- Since integral noise warping does not prevent noticeable inconsistency in the details in the frames, is it possible to enhance your method by adding an explicit temporal consistency term between frames?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The new interpretation/formulation of integral noise warping is novel and leads to theoretically backed relation to Brownian bridges, whose Markovian property allows to derive convenient implementation methods.\n\n- The derived methods are useful in practical applications where image-based diffusion models are used for video generation in a training-free manner.\n\n- The paper is well-written.", "weaknesses": "- There is still inconsistency in the details in the frames.  \n\n- The quantitative SDEdit-based video results of the method (and HIWYN) are similar to the competitors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBcE5XqefH", "reviewer": "Reviewer_gG1q", "summary": "The paper presents an advanced noise manipulation method to enhance temporal consistency in video generation using diffusion models. The authors address the computational inefficiency in Chang et al.'s (2024) method, which ensures temporal consistency but requires high-resolution upsampling. By shifting from a Lagrangian to an Eulerian perspective and using Brownian bridge increments instead of high-resolution upsampling, the approach achieves the same precision while significantly improving speed and memory efficiency. The author also introduces two methods for calculating noise overlap areas—grid-based and particle-based. The particle-based approach is faster, more memory-efficient, and more robust to non-injective mappings. Results show that this method not only preserves Gaussian noise properties but also achieves speed improvements by several orders of magnitude, maintaining or even improving temporal consistency in video applications, making it more practical for real-world scenarios.", "questions": "1. The symbols are not clearly defined, for example, in the description of the Eulerian Procedure on lines 158-166, $M, m_k, l_k ...$ which can lead to difficulties in reading and understanding.\n\n2. The ultimate computation goal is the same as in HIWYN: $ \\tilde I_{i,j}$ , as shown in Figure 1, the goal is to obtain the \"area\" of the red region $ \\tilde A_{i,j}$ . However, from the Eulerian View, it is evident that $ \\tilde A_{i,j} $ will be dispersed across each $ A_{i,j}$ ; in the example in Figure 2, this accounts for 39%. $ I_{i,j}$ can be considered a Brownian bridge from 0 to $ I_{i,j} $, and the area of each deformed region can be viewed as \"time.\" Using this, it might be possible to directly sample the 39% value of the Brownian bridge and then gather it with other red parts across various $ A_{i,j} $ regions to obtain the final $ \\tilde A_{i,j}$ . I am not sure if my understanding is correct. If this understanding is correct, then the differently colored regions within $A_{i,j}$ should originate from sampling with different conditions (as indicated by equation (1)), meaning that each color belongs to a different distribution. Would it then be valid to interpret this as an addition of values from Brownian bridges?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- By using Brownian bridge increments instead of high-resolution upsampling, the proposed method achieves efficient noise manipulation without sacrificing accuracy, greatly enhancing speed and memory efficiency.\n- The method maintains temporal consistency of noise in video generation, avoiding flickering or inconsistencies between frames seen in traditional methods, resulting in smoother and more natural video outputs.\n- The particle-based noise overlap calculation method is not only faster but also more robust to non-injective mappings, making it suitable for complex, non-linear real-world scenarios and offering high application potential.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwZJYpCIDF", "reviewer": "Reviewer_XEe5", "summary": "This paper aims to tackle the computational inefficiency of the noise warping algorithm first presented by Chang et al. [1]. To this end, the authors propose an Eulerian perspective of the warping problem, which allows achieving equivalent results with orders of magnitude speed-up. The algorithm scatters increments of Brownian bridges from the original pixels to the warped ones, effectively removing the need for a high-resolution sample, as it simulates the case of infinite resolution. Additionally, the paper proposes a particle-based approach which is simpler to implement, more robust and even faster, at the cost of minor loss in accuracy in real-world scenarios. To support the practical value of the proposed method, the authors show applications with video translation tasks and 3D editing.\n\n[1] How I Warped Your Noise: a Temporally-correlated noise prior for diffusion models. (Chang et al.)", "questions": "Not many questions, but I’m curious to know if the authors have any idea what practical use cases could actually benefit from the almost real-time version of noise warping they proposed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. While the link between noise warping and brownian bridges was noted in previous work, the insights provided by the Eulerian perspective to leverage this connection are novel and theoretically sound.\n\n2. The proposed algorithms greatly improve the computational efficiency of the problem compared to previous work, which can open the door for more potential applications.\n\n3. The particle-based algorithm tackles another limitation of previous work, which was limited to diffeomorphic warpings. This simpler approach can make such method more accessible.\n\n4. The paper is well-written and clear in its explanations, and the proposed method is well validated.", "weaknesses": "1. It is hard to tell from the supplementary video results whether noise warping is actually improving coherency, especially in the 3D case.\n2. While the noise warping method is now faster, the authors do not show what practical benefits this speed-up can bring (e.g. what new applications does this open the door to), which slightly hinders the significance of this work.\n3. As mentioned by the authors, the method is still limited by the accuracy of the provided flow, and the effects of noise warping in video translation tasks remain based on empirical observations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)