Infinite-Resolution Integral Noise Warping for Diffusion Models

## Abstract

Abstract Adapting pretrained image-based diffusion models to generate temporally consistent videos has become an impactful generative modeling research direction.
Training-free noise-space manipulation has proven to be an effective technique, where the challenge is to preserve the Gaussian white noise distribution while adding in temporal consistency.
Recently, Chang et al. ( 2024 ) formulated this problem using an integral noise representation with distribution-preserving guarantees, and proposed an upsampling-based algorithm to compute it. However, while their mathematical formulation is advantageous, the algorithm incurs a high computational cost.
Through analyzing the limiting-case behavior of their algorithm as the upsampling resolution goes to infinity, we develop an alternative algorithm that, by gathering increments of multiple Brownian bridges, achieves their infinite-resolution accuracy while simultaneously reducing the computational cost by orders of magnitude.
We prove and experimentally validate our theoretical claims, and demonstrate our method’s effectiveness in real-world applications. We further show that our method readily extends to the 3-dimensional space.

## 1 Introduction

The success of diffusion models in image generation and editing (Rombach et al., ; Nichol et al., ; Ho et al., ; Zhang et al., ) has spurred significant interest in lifting these capacities to the video domain (Singer et al., ; Durrett, ; Gupta et al., ; Blattmann et al., ; Ho et al., ; Guo et al., ).
While training video diffusion models directly on spatiotemporal data is a natural idea, practical concerns such as limited availability of large-scale video data and high computational cost have motivated investigations into training-free alternatives. One such approach is to use pre-trained image models to directly generate video frames, and utilize techniques such as cross-frame attention, feature injection and hierarchical sampling to promote temporal consistency across frames (Ceylan et al., ; Zhang et al., ; Khachatryan et al., ; Cong et al., ).

Among these techniques, the controlled initialization of noise has been consistently shown to be an important one (Ceylan et al., ; Khachatryan et al., ). However, most existing approaches for noise manipulation either compromise the noise Gaussianity (and subsequently introduce a domain gap at inference time), or are restricted to simple manipulations such as filtering and blending which are insufficient for capturing complex temporal correlations.
Recently, Chang et al. ()
proposed a method that both preserves Gaussian white noise distribution and well captures temporal correlations via *integral noise warping*: each warped noise pixel integrates a continuous noise field over a polygonal deformed pixel region, which is computed by summing subpixels of an upsampled noise image. However, their method’s theoretical soundness and effectiveness are followed by its high-end computational cost in both memory and time, which not only incurs a significant overhead at inference time but also limits its useability in novel applications (Kwak et al., ).

In this paper, we introduce a new noise-warping algorithm that dramatically cuts down the cost of Chang et al. () while fully retaining its virtues. Our key insight for achieving this lies in that, when adopting an Eulerian perspective (as opposed to the original Lagrangian one), the limiting-case algorithm of Chang et al. () for computing a warped noise pixel reduces to summing over increments from multiple Brownian bridges (Durrett, , Section 8.4).
In place of the costly upsampling procedure, sampling the increments of a Brownian bridge can be done efficiently in an autoregressive manner (2).
We build upon this to devise the *infinite-resolution integral noise warping* algorithm (1)
which directly resolves noise transport in the continuous space, when given an oracle that returns the overlapping area between a pixel square and a deformed pixel region (Section 2.3).

We propose two concrete ways to compute this oracle, leading to a grid-based and a particle-based variant of our method. Similar to Chang et al. (), the grid-based variant (Algorithm 2) computes the area by explicitly constructing per-pixel deformed polygons, and is exactly equivalent to the existing approach (Chang et al., ) with an infinite upsampling resolution, while running $8.0\times$ to $19.7\times$ faster and using $9.22\times$ less memory(^1^11Since the official code of Chang et al. () is not available, performance is compared using our reimplementation in Taichi (Hu et al., ), which we find to be faster than as reported in the original paper.). Inspired by hybrid Eulerian-Lagrangian fluid simulation (Brackbill et al., ), our novel particle-based variant (Algorithm 3) computes area in a fuzzy manner, which not only offers a further $5.21\times$ speed-up over our grid-based variant, but is also agnostic to non-injective maps. In real-world scenarios, the particle-based variant shows no compromise in generation quality compared to the grid-based one (see video results), while offering superior robustness, efficiency, simplicity, and extensibility to higher dimensions.

In summary, we propose a new noise-warping method to facilitate video generation by lifting image diffusion models. Through analyzing the limiting case of the current state-of-the-art method (Chang et al., ) with an infinite upsampling resolution, we derive its continuous-space analogy, which fully retains its distribution-preserving and temporally-coherent properties, while achieving orders-of-magnitude speed-up, warping $1024\times 1024$ noise images in $\sim 0.045$s (grid variant) and $\sim 0.0086$s (particle variant) using a laptop with a Nvidia RTX 3070 Ti GPU.

Figure: Figure 1: When the image grid deforms, the Lagrangian view tracks a deformed pixel region, while the Eulerian view tracks the undeformed pixel square as it gets partitioned into multiple regions. On the right, we leverage the exchangeability of upsampled subpixels to convert the Lagrangian gathering procedure into scattering noise subpixels to overlapped deformed pixel regions.
Refer to caption: x1.png

## 2 Methodology

In this section, we introduce our method as follows:

- •
We present an equivalent Eulerian interpretation (Figure 1) for the method by Chang et al. (), which was developed from a Lagrangian viewpoint.
- •
We show that the limiting algorithm of the Eulerian formulation as upsampling level goes to infinity is equivalent to sampling increments of Brownian bridges.
- •
We present our main algorithm (1)
which, given a partition record that returns the overlapping area between a pixel square and a deformed pixel region, samples increments of Brownian bridges and scatters the increments to form the warped noise image.
- •
We propose two concrete algorithms for computing the overlap areas. The grid-based Algorithm 2 extends Chang et al. () to infinite resolution without the overhead of upsampling.
The particle-based Algorithm 3 departs from grid-based discretization and uses particles instead, resulting in a simpler algorithm that is robust to degenerate maps.

### 2.1 Noise Warping: an Alternative Eulerian Perspective

Given a $D\times D$ prior
noise image $I_{W}\in\mathbb{R}^{D\times D}$(^2^22 Here we assume that the noise image is square and has a single channel only to simplify notation. In practice, the noise image can have arbitrary aspect ratio and number of independent channels.) and a deformation map $\psi:[0,1]^{2}\to[0,1]^{2}$, the noise-warping algorithm (Chang et al., ) computes the warped noise image $\widetilde{I}_{W}\in\mathbb{R}^{D\times D}$ with upsampling level $N\in\mathbb{Z}_{\geq 1}$ as follows:

- 1.
For $i,j=1,\ldots,D$, upsample noise pixel $[I_{W}]_{i,j}$ to an $N\times N$ subimage $[\widehat{I}_{W}]_{i,j}\in\mathbb{R}^{N\times N}$:
$\displaystyle[\widehat{I}_{W}]_{i,j}=\frac{[I_{W}]_{i,j}}{N^{2}}+\frac{1}{N}\!
\left(\!Z\!-\!\frac{S}{N^{2}}\!\right)\!,\text{ with }Z\sim\mathcal{N}(\mathbf
{0},\mathbf{I})\text{ and }S=\sum\nolimits_{k=1}^{N^{2}}Z_{k}.$
(1)
The subimage for each pixel assembles into an $ND\times ND$ upsampled noise image $\widehat{I}_{W}$.
- 2.
For $i,j=1,\ldots,D$, the pixel square $A_{i,j}\coloneq[\frac{i-1}{D},\frac{i}{D}]\times[\frac{j-1}{D},\frac{j}{D}]$ is warped to a deformed pixel region $\widetilde{A}_{i,j}\coloneq\psi(A_{i,j})$, and the warped noise pixel $[\widetilde{I}_{W}]_{i,j}$ is set to be the sum of all subpixels in $\widehat{I}_{W}$ covered by $\widetilde{A}_{i,j}$ divided by $\sqrt{|\widetilde{A}_{i,j}|}$, where $|A|$ denotes the Lebesgue measure of a Borel set $A\subset\mathbb{R}^{2}$.

We describe an alternative but equivalent procedure by making the following two observations, which are illustrated in Figure 1.

Gathering Noise $\rightarrow$ Scattering Noise.
While the original procedure computes the warped noise image by *gathering* the upsampled noise subpixels in each deformed pixel region $\widetilde{A}_{i,j}$ in a *Lagrangian* fashion, we can instead use an alternative procedure by *scattering* the upsampled noise subpixels in each pixel square $A_{i,j}$ to overlapping deformed pixel regions.
This new *Eulerian* procedure does not change the output, but it yields new insights in conjunction with our second observation.

Scattering Noise $\rightarrow$ Counting Overlapping Subpixels.
Observe that the $N\times N$ subpixels in $[\widehat{I}_{W}]_{i,j}$, for every $i,j$, are correlated only through their sum $S$ when conditioning on $[I_{W}]_{i,j}$ (1), so they are exchangeable.
Hence, when scattering these upsampled noise subpixels to deformed pixel regions, the order of scattering does not matter, and we only need to count *the number of subpixels* covered by each deformed pixel region.

Alternative Eulerian Procedure.
Putting both observations together, we now describe an alternative procedure to Chang et al. () with unaltered output:

- 1.
For each noise image pixel $[I_{W}]_{i,j}$, draw an upsampled subimage, now represented as a 1D vector $X\in\mathbb{R}^{N^{2}}$ using (1). Then, compute a prefix sum $H_{i,j}$ via $[H_{i,j}]_{k}\coloneq\sum_{q=1}^{k}X_{q}$ for $k=1,\ldots,N^{2}$.
- 2.
Warp each pixel square and compute deformed pixel regions $\widetilde{A}_{i,j}$ as before.
- 3.
For each $A_{i,j}$, identify all $M$ deformed pixel regions $\{\widetilde{A}_{\ell_{k},m_{k}}\}_{k=1,\ldots,M}$ that overlap with $A_{i,j}$. Form $L\in\mathbb{Z}_{\geq 0}^{M}$ where $L_{k}$ represents the number of upsampled subpixels covered by the $k^{\text{th}}$ overlap. Then, compute a prefix sum $[C_{i,j}]_{k}\coloneq\sum_{q=1}^{k}L_{q}$. For $k=1,\ldots,M$, accrue $[H_{i,j}]_{[C_{i,j}]_{k}}-[H_{i,j}]_{[C_{i,j}]_{k-1}}$ to $[\widetilde{I}_{W}]_{\ell_{k},m_{k}}$, the $k^{\text{th}}$ overlapped warped noise pixel.
- 4.
Divide each warped noise pixel $[\widetilde{I}_{W}]_{i,j}$ by $\sqrt{|\widetilde{A}_{i,j}|}$.

Discussion.
Compared to the original procedure by Chang et al. (), this alternative but equivalent algorithm highlights how the upsampled subpixels of $[I_{W}]_{i,j}$ are scattered to form the warped noise pixels.
In particular, each warped noise pixel receives the *sum of a continuous segment* in $H_{i,j}$.
Since $H_{i,j}$ is a summation of weakly correlated and exchangeable subpixels, once conditioned on $[I_{W}]_{i,j}$, *can we avoid explicitly instantiating every single subpixel*, but instead model the *sum* of these weakly correlated subpixels?

The key insight of this paper is that when the upsampling resolution $N\to\infty$, the scaling limit of the prefix sum $H_{i,j}$ (with proper interpolation and time scaling to a continuous function) is precisely the Brownian bridge (Durrett, , Section 8.4) conditioned on $[I_{W}]_{i,j}$.
Once this connection is established, it is easy to progressively sample increments of the Brownian bridge, resulting in a clean and efficient noise-warping algorithm that bypasses the need for upsampling in Chang et al. ().

Figure: Figure 2: Connection between Eulerian noise-warping and increments of a Brownian bridge for a fixed prior noise pixel $[I_{W}]_{i,j}$. The overlapping area of each colored warped region becomes the time increment for the Brownian bridge. Hence, sampling the Brownian bridge at these times and taking consecutive differences yields integral noise that is scattered to form each warped noise pixel.
Refer to caption: x2.png

### 2.2 Infinite-Resolution Noise Scattering

In this section, we first derive a scaling limit result to Brownian bridges. We then illustrate that the limiting version of the Eulerian procedure from the previous section matches precisely this scaling limit result.
Lastly, we describe an autoregressive way to sample increments of a Brownian bridge that is linear in runtime in terms of the number of increments.

###### Theorem 1 (Scaling limit to Brownian bridge) .

Let $\{Z_{n}\}$ be a sequence of i.i.d. random variables with finite variance that are normalized such that $\mathbb{E}[Z_{n}]=0$ and $\mathrm{Var}(Z_{n})=1$.
For $c\in\mathbb{R}$, define

$$ $\displaystyle S_{n}\coloneq\sum_{i=1}^{n}Z_{i},\qquad X_{i,n}\coloneq\frac{c}{ n}+\frac{1}{\sqrt{n}}\left(Z_{i}-\frac{S_{n}}{n}\right).$ $$

Consider the sequence of random continuous functions $\{H_{n}(t)\}\subset C[0,1]$ defined as

$$ $\displaystyle H_{n}(t)\coloneq\sum_{i=1}^{\lfloor nt\rfloor}X_{i,n}+(nt- \lfloor nt\rfloor)X_{\lfloor nt\rfloor+1,n}.$ $$

Then the sequence $\{H_{n}\}$ converges in distribution under the sup-norm metric on $C[0,1]$ to
$B_{c}(t)\coloneq W(t)-tW(1)+tc$,
the Brownian bridge ending at $c$, where $W(t)$ is standard Brownian motion.
Moreover, in distribution, we have
$B_{c}(t)\stackrel{{\scriptstyle d}}{{=}}(W(t)\mid W(1)=c),$
where $(W(t)\mid W(1)=c)$ is the disintegrated measure (Pachl, ) of $W(t)$ on $W(1)=c$.

We prove Theorem 1 in Appendix A.
To connect the Eulerian procedure with the setup in Theorem 1, let us fix a pixel $[I_{W}]_{i,j}$, and let $B\coloneq B_{[I_{W}]_{i,j}}$, $H\coloneq H_{i,j}$, $C\coloneq C_{i,j}$ to simplify the notation. By setting $n=N^{2}$ and $c=[I_{W}]_{i,j}$, the sequence $\{X_{k,n}\}$ from the theorem has exactly the same law as the upsampled subpixels in $[\widehat{I}_{W}]_{i,j}$.
Moreover, $H_{nt}=H_{n}(t)$ when $nt\in\mathbb{Z}_{\geq 1}$.
By taking $N\to\infty$, implying $n\to\infty$, for any $t_{1},\ldots,t_{M}\in[0,1]$, we have the convergence in distribution of
$\left(H_{\lfloor n{t_{1}}\rfloor},\ldots,H_{\lfloor n{t_{M}}\rfloor}\right)
\stackrel{{\scriptstyle d}}{{\to}}\left(B(t_{1}),\ldots,B(t_{M})\right)$.
Recall in the Eulerian procedure, we only need to access the prefix sum $H$ at indices $\{C_{k}\}_{k=1}^{M}$, where $C_{k}$ counts the number of upsampled subpixels covered by the first $k$ overlaps.
This suggests that if we choose

$$ $\displaystyle t_{k}=\lim_{N\to\infty}\frac{C_{k}}{N^{2}}=\sum_{k^{\prime}=1}^{ k}\left|A_{i,j}\cap\widetilde{A}_{\ell_{k^{\prime}},m_{k^{\prime}}}\right|,$ $$

and use $B(t_{k})$ in place of $H_{k}$,
then we just need to sample from $B$ at times $t_{1},\ldots,t_{M}$ — precisely the limiting algorithm of the Eulerian procedure.
We illustrate this connection in Figure 2.

Autoregressive Sampling of Brownian Bridges.
Since a Brownian bridge is a Markov process (Oksendal, , Exercise 5.11), we can sample the vector $(B_{c}(t_{1}),\ldots,B_{c}(t_{M}))$ in an autoregressive fashion, each time sampling $B_{c}(t_{k+1})$ conditioned on $B_{c}(t_{k})$:

$$ $\displaystyle\left(B_{c}(t_{k+1})\mid B_{c}(t_{k})=q\right)\stackrel{{ \scriptstyle d}}{{=}}\operatorname{\mathcal{N}}\left(\frac{1-t_{k+1}}{1-t_{k}} q+\frac{t_{k+1}-t_{k}}{1-t_{k}}c,\frac{(t_{k+1}-t_{k})(1-t_{k+1})}{1-t_{k}} \right).$ (2) $$

Once the Brownian bridge at times $t_{k}$ is sampled, we just need to accrue the increments $B_{c}(t_{k})-B_{c}(t_{k-1})$ to $[\widetilde{I}_{W}]_{\ell_{k},m_{k}}$, the $k^{\text{th}}$ overlapped warped noise pixel.
This allows us to present Algorithm 1.
Compared to the discrete procedures described earlier, we no longer need upsampling. In addition, we exploited the autoregressive nature of Brownian bridges to bring down the time complexity to linear in the number of overlapping warped pixel regions.

Figure: Algorithm 1 Infinite-Resolution Integral Noise Warp

Figure: Figure 3: The grid-based variant (left) computes the overlapping areas by explicitly constructing the polygon for the deformed pixel region. The particle-based variant (middle) approximates these areas with a weighting kernel. With degenerate maps (right), the fixed topology of the grid-based variant can lead to problems, while the connectivity-free, particle-based variant remains stable.
Refer to caption: x3.png

Preservation of Gaussian White Noise.
A central desideratum of noise warping is that the resulting warped noise image $\widetilde{I}_{W}$ needs to have pixels that are i.i.d. standard Gaussians when the prior noise image $I_{W}$ is Gaussian white noise.
This ensures that the warped noise is in-distribution for a pre-trained diffusion model.
Our algorithm automatically guarantees this preservation of Gaussianity, as long as the warping function $\psi$ is injective.
To see this, the injectivity of $\psi$ implies that the warped pixel regions are non-overlapping in the square $[0,1]^{2}$.
For each $A_{i,j}$, since $[I_{W}]_{i,j}\stackrel{{\scriptstyle d}}{{=}}\mathcal{N}(0,1)\stackrel{{
\scriptstyle d}}{{=}}W(1)$, by the conditional interpretation of Brownian bridges (1), when marginalizing out $[I_{W}]_{i,j}$, the Brownian bridge $B_{[I_{W}]_{i,j}}$ reduces to standard Brownian motion.
Since the increments of the Brownian motion are independent Gaussians, the contribution to a deformed pixel region is simply a zero-mean Gaussian with variance equal to the overlapping area.
Therefore, each deformed pixel region will receive the sum of a number of independent Gaussians whose variances sum to the area of the region.
The scaling by the inverse square root of the area in Algorithm 1 thus makes each warped noise pixel an i.i.d. standard Gaussian.

###### Theorem 1 (Scaling limit to Brownian bridge) .

### 2.3 Building Partition Records

To compute Algorithm 1, we need a way to compute the partition record $\mathcal{P}$, which specifies how each pixel square is partitioned by multiple deformed pixel regions.
In this section, we present one grid-based and one particle-based method for building $\mathcal{P}$. In particular, for each pixel square with indices $(u,v)$, we compute $\mathcal{P}_{u,v}$ as a list of 3-tuples $(a,i,j)$, where $(i,j)$ identifies the overlapped deformed pixel region and $a$ represents the overlapping area. Both variants are illustrated in Figure 3.

Figure: Algorithm 2 Grid-based Partition

Figure: Algorithm 3 Particle-based Partition

Our grid-based method (Algorithm 2 and Figure 3, left) follows Chang et al. () by modeling each deformed pixel region as an octagon and computes overlapping areas by clipping it aginast undeformed pixel squares. Our particle-based method (Algorithm 3 and Figure 3, middle) borrows from the grid-to-particle techniques in fluid particle-in-cell methods (Brackbill et al., ), where we treat each deformed pixel region as a particle and each undeformed pixel square as a grid cell. Each particle requests area from nearby cells based on distance; upon receiving requests, each cell normalizes the requests to ensure partition-of-unity, and distributes its area to contacting particles.

Discussion.
Conceptually, our grid and particle-based methods correspond to two different interpretations of $\psi$ when provided as discrete samples (e.g., an optical flow image). The grid-based method implicitly reconstructs the continuous $\psi$ field by linear interpolation, whereas the particle-based method assumes $\psi$ is only known point-wise. The implication is that when $\psi$ is smooth, linear interpolation works well and the grid-based method will yield a higher-quality warp as seen in Figure 10. But when $\psi$ is non-smooth, which is commonly the case in real world, linear interpolation can lead to degenerate polygons as illustrated on the right of Figure 3. The spurious overlaps between the degenerate polygons will lead to spatial correlation in the warped noise image.
Although both Chang et al. () and our grid-based method implement fail-safes(^3^33In our case, we clamp the input $t_{k+1}$ to $1$ before sampling by 2. Intuitively, this means that when an undeformed pixel has assigned its entire pixel region to deformed pixels, later noise requests will be neglected.) to avoid noise sharing and maintain spatial independence in practice, they suffer from the intrinsic ambiguity caused by these overlaps. On the other hand, the particle-based method circumvents such overlaps to begin with.

In addition, we highlight the simplicity and parallelizability of the particle-based method, as it boils down the computation of $\mathcal{P}$ to evaluating one bilinear kernel per pixel. Leveraging this fact, we can conveniently and efficiently extend our noise warping algorithm to higher spatial dimensions by replacing the bilinear kernel to its higher-dimensional counterparts, as shown in Figure 6.

## 3 Results

In this section, we verify our theoretical claims by showing that our both variants preserve Gaussian white noise distribution, and that Chang et al. () (HIWYN) converges to our grid-based variant as $N$ increases.
We analyze the behaviors of our grid-based and particle-based variants under diffeomorphic and non-diffeomorphic deformations.
We then apply our method in video generation and benchmark against existing methods (Ge et al., ; Chen et al., ; Chang et al., ).
Finally, we extend our method to warping volumetric noise and demonstrate a use case in 3D graphics.

Figure: Figure 4: Preservation of Gaussian white noise achieved by different warping methods. We report scores and p-values for both Moran’s $I$ (spatial correlation) and K-S test (normality). We show that results from our method (both variants) and HIWYN are indistinguishable from white Gaussian noise, while generic warping methods lead to corrupted noise.
Refer to caption: x4.png

Figure: Figure 5: Convergence of HIWYN to our method as $N$ increases. Top left: experimental setup with prior noise and deformation map. Top middle: 2-Wasserstein distance $W^{N}$ between the output of HIWYN and ours. Top right: statistics table. Bottom: $W^{N}$ difference image between the output of HIWYN and ours as $N$ increases. Notice $W^{N}$ becomes statistically insignificant for $N\geq 64$.
Refer to caption: x5.png

Gaussian White Noise Preservation.
In Figure 4,
we iteratively warp a noise image
by the same deformation map for 50 timesteps.
We gauge the output noise’s resemblance to Gaussian white noise by measuring normality using one-sample Kolmogorov-Smirnov (K-S) test and detecting spatial correlation using Moran’s $I$.
Our results show that both HIWYN and ours generate noise images indistinguishable from Gaussian white noise while baseline warping methods cannot pass either test.

Convergence of Chang et al. ().
We validate that our method (grid-based) is the limiting case of HIWYN.
Starting with an $8\times 8$ prior noise image and a flow map (Figure 5, top left), we run our method and HIWYN with upsampling resolutions $N\in\{2,4,8,\ldots,256\}$ for \num100000 independent runs to estimate the warped noise’s distribution.
For each $N$, we compute the
2-Wasserstein distance $W^{N}$ between the output distribution of HIWYN and that of our method.
The results in Figure 5 demonstrate the convergence of HIWYN to our method as $N$ increase, and reveal that $N\!=\!8$ (recommended by Chang et al. ()) is not yet in the converged phase to yield a negligible $W^{N}$.

Performance Comparison.
For our method (both variants) and HIWYN with upsampling levels $N\in\{2,4,8\}$, we perform 100 independent runs on a $1024\times 1024$ image.
We report the kernel time
with CPU and GPU backends (Figure 7) as well as the memory usage. The runtime and memory usage of both our variants are largely comparable to those of HIWYN with $N\!=\!2$.
Compared to HIWYN with $N\!=\!8$, both our variants offer order-of-magnitude improvements in runtime and memory usage.
Specifically, our grid-based variant extends HIWYN to infinite upsampling resolution while being $19.7\times$ faster on CPU and $8.0\times$ faster on GPU, using $9.22\times$ less memory; and our particle-based method, albeit not strictly equivalent to HIWYN at $N\!=\!\infty$,
achieves a $41.7\times$ speedup on GPU.
In the following sections, we show that our particle-based variant consistently achieves comparable quality to the grid-based variant in real-world scenarios (see video results).

Comparison between Grid-Based and Particle-Based Variants.
In Figure 10, we compare both variants when the deformation map is diffeomorphic under different levels of distortion.
Visually, the difference between the two variants is negligible at frame 25 and becomes noticeable at frame 100. We measure this difference by comparing the deformed regions for each pixel in terms of IoU and weighted Chamfer distance.
We additionally compare the particle-based result with that of an identity-map baseline (right column in Figure 10), which shows that the gap between the two variants remains small even under large distortion.
In Figure 8, we stress test both variants under non-diffeomorphic maps obtained using optical flow (Teed & Deng, ) on a real-world video (Brox & Malik, ). In images 3 and 4, we see that the real-world flow map induces inverted meshes for the grid-based variant and clustered particles for the particle-based variant. While clustered particles are guaranteed to be assigned disjoint regions as prescribed by Algorithm 3, mesh inversions cause noise contention issues due to polygon overlaps. In images 5 and 6, we mark the grid cells with noise contention in orange, which occurs in the grid-based variant but not in the particle-based variant.

Conditional Video Generation.
We apply our method to conditional video generation by adapting SDEdit (Meng et al., ), a conditional image generation method, to produce temporally consistent video frames.
We apply Perturbed-Attention Guidance (Ahn et al., ) to the unconditional models with scale 3.0. Our two inputs are a conditioning video (generated by applying a median filter to real-world videos following Chen et al. ()) and an optical flow field estimated using RAFT (Teed & Deng, ).
Without noise manipulation, if we run SDEdit frame-by-frame (Figure 9, bottom row), the details (e.g., in the tower and trees) would result in strong flickering.
By warping the noise using the optical flow, the temporal consistency is much improved.
As shown in Figure 9, our methods (both variants) and HIWYN yield comparable visual qualities.
Full experiments that shows comparison with Control-A-Video (Chen et al., ) and PYoCo (Ge et al., ) and additional baselines are provided in Figures B.3 and B.4 with generation quality metrics reported in Table 1.
We refer to our supplementary video for better visualization.

Figure: Figure 6: 3D noise warped by our particle variant.
Refer to caption: x6.png

3D Noise Warp.
We extend our particle-based algorithm
to 3D by replacing the bilinear kernel
with a bicubic kernel in Algorithm 3 and apply it to GaussianCube (Zhang et al., ), which denoises a dense 3D noise grid to reconstruct 3D Gaussians.
We adapt it to perform conditional generation a la SDEdit. Starting with a 3D pickup truck generated unconditionally, we condition the model to generate vehicles with smaller and larger cabins by deforming the truck with a horizontal shear velocity field. We compare the results from using random noise to those using noise warped with our particle-based method. Using the warped noise improves the consistency, reducing the flickering of the cars’ geometries and textures. We show the results in Figure B.1 and refer to our supplementary video for better visualization.

Figure: Figure 7: Runtime and memory usage of our method vs. HIWYN with $N\!=\!2,4,8$. We compare total allocated memory and kernel time on a CPU/GPU. The computation is done on a laptop with Intel i7-12700H and Nvidia RTX 3070 Ti.
Refer to caption: extracted/5972604/real_figures/bar_chart_revised.png

Figure: Figure 8: Comparison of grid-based vs. particle-based variants under non-diffeomorphic optical flow. Pixels with detected overlaps are colored in orange. Further results are given in Figure B.2.
Refer to caption: x7.png

Figure: Figure 9: We compare the temporal consistency with different noise initialization schemes. We obtain the image sequence using SDEdit with the conditional signal shown at the top. The results of our method (both variants) and HIWYN are shown here, and full results with additional baselines and benchmarks are shown in Figure B.3. We highlight the details of the tower with the inset.
Refer to caption: x8.png

Figure: Figure 10: Comparison between the grid-based and particle-based variants for building partition records when the deformation map is diffeomorphic. The first column shows the deformation map at different frames. The second and third columns visualize warped pixel regions by the two variants. The fourth and fifth columns show IoU (larger is better) and Chamfer distance (smaller is better) between these deformed pixel regions. In the two plots on the right, we contextualize and visualize how much the two variants differ by plotting their IoU and Chamfer distance against those computed between the grid-based variant and the identity map, which shows that the particle-based variant remains close to the grid-based variant even under large distortion.
Refer to caption: x9.png

## 4 Related Works

Noise in Diffusion Models.
Diffusion models generate images from input noise, and noise can thus be considered the counterpart to the latent codes utilized in GAN models. As such, the outputs of diffusion models have dependencies and correlations to the initial input noise, making noise a useful handle to control temporal consistency (Khachatryan et al., ).
In addition to Chang et al. () which this work was inspired by and improves upon, there are various other temporal noise manipulation techniques that do not preserve Gaussian noise distribution– some methods (Ma et al. (); Ren et al. ()) blend high frequency Gaussian noise with low frequency motion, while others (Mokady et al. (); Wallace et al. ()) rely on approximating the inversion of noise from temporally coherent image sequences. Pandey et al. () goes one step further and manipulates inverted noise in 3D space. These approaches are flexible but degrade the output of the diffusion model due to the domain gap between inference time noise and training time noise, and as such, have occasionally been accompanied by mitigation strategies such as anisotropic diffusion (Yu et al. ()).
Noise manipulation is also not limited to the generation and stylization of videos, but has various applications in image editing (Hou et al. (); Pandey et al. ()) and 3D mesh texturing (Richardson et al. ()) as well.

Noise in Computer Graphics. While our noise warping work draws main inspiration from simulation techniques, spatial noise manipulation has been extensively studied in the graphics community through applications in animation and rendering. Works like (Kass & Pesare, ; Burley et al., ) present 2D noise manipulation techniques that add a stylized organic hand-drawn look to computer-generated animation via dynamic noise textures (Perlin, ). In order to make sure the stylization is temporally consistent and visually pleasing, noise textures are deformed in a way that makes them consistent with the underlying animation, but little emphasis is given to the preservation/rigor of the noise distribution. On the other hand, properties of 2D spatial noise have been extensively and rigorously studied in rasterization and raytracing literature (Cook, ; Lagae & Dutré, ), originating from the idea of using dithering to reduce banding and quantization artefacts in image signal processing (Roberts, ). In particular, the lack of low frequency details and clumping in blue noise as opposed to white Gaussian noise has made it the choice of foundational antialiasing methods such as Poisson disc sampling (McCool & Fiume, ), and recent progress made in this line of antialiasing research has close ties with our methodology. For example,
Wolfe et al. () look at accelerating rendering tasks by extending spatial blue noise to the temporal domain, while Huang et al. () show promising results in supplementing white noise with blue noise during diffusion model training.

## 5 Conclusions

In this paper, we presented infinite-resolution integral noise warping, a novel algorithm for computing temporally coherent, distribution-preserving noise transport to guide diffusion models into generating consistent results. By deriving a continuous-space analogy to the discrete, upsampling-based strategy of the current state-of-the-art (Chang et al., ), our method not only further improves the accuracy by effectively raising the upsampling resolution to infinity, but also drastically reduces the computational cost, processing high-resolution noise images in real-time, which removes its main limitation. We also highlight the connotations of our new perspective beyond the performance gains, as it facilitates agnosticism to non-injective maps and extensibility to higher dimensions.

Our work may be extended in a few directions. First, our particle-based variant does not capture temporal correlations induced by contraction or expansion, which may be addressed in the future with Voronoi partitioning.
Secondly, although we only show use cases that leverage flow maps for temporal consistency, our method can operate on other map types such as UV maps for 3D consistency, which might be explored in future works. Thirdly, the connection between the consistency of the initial noise and that of the generated results remains empirical and invites theoretical justifications. Finally, the efficacy of noise warping for latent diffusion models remains to be investigated.

## Acknowledgement

We thank Lukas Lepicovsky, Ioan Boieriu, David Michielsen, Mohsen Mousavi, and Perry Kain from Eyeline Studios, for providing data that kickstarted this project and for assisting in shaping our research with practical future use cases. We also thank Austin Slakey for sharing his insights on training diffusion models, and Tianyi Xie for filming his cat for our testing.

## Appendix A Proof of Theorem 1

###### Proof.

By unrolling the definitions, for $t\in[0,1]$, we have

$$ $\displaystyle H_{n}(t)=S_{n}^{*}(t)-tS_{n}^{*}(1)+tc,\qquad S_{n}^{*}(t) \coloneq\frac{1}{\sqrt{n}}\left(\sum_{i=1}^{\lfloor nt\rfloor}Z_{i}+(nt- \lfloor nt\rfloor)Z_{\lfloor nt\rfloor+1}\right).$ $$

By Mörters & Peres (, Theorem 5.22), $\{S_{n}^{*}\}_{n\in\mathbb{Z}_{\geq 1}}$ converges in distribution to $W(t)$ under the sup-norm metric of $C[0,1]$.
To lift this convergence to the sequence $\{H_{n}\}_{n\in\mathbb{Z}_{\geq 1}}$, observe that the function $g:C[0,1]\to C[0,1]$ defined by

$$ $\displaystyle g(x(t))\coloneq x(t)-tx(1)+tc$ $$

is continuous under the sup-norm metric.
To verify this, suppose $\lim_{n\to\infty}f_{n}=f$ for $\{f_{n}\}_{n\in\mathbb{Z}_{\geq 1}},f\in C[0,1]$. Then

$$ $\displaystyle\|g(f_{n})-g(f)\|_{\infty}$ $\displaystyle=\sup_{t\in[0,1]}\left|(f_{n}(t)-tf_{n}(1)+tc)-(f(t)-tf(1)+tc)\right|$ $\displaystyle\leq\|f_{n}-f\|_{\infty}+\|f_{n}(1)-f(1)\|\leq 2\|f_{n}-f\|_{ \infty}\to 0.$ $$

Hence, by the continuous mapping theorem,

$$ $\displaystyle g(S_{n}^{*})=H_{n}\stackrel{{\scriptstyle d}}{{\longrightarrow}} B(t)-tB(1)+tc.$ $$

To show

$$ $\displaystyle W(t)-tW(1)+tc=(W(t)\mid W(1)=c),$ $$

first of all, the conditioning $(W(t)\mid W(1)=c)$ is interpreted as the limit of $(W(t)\mid\left|W(1)-c\right|<\epsilon)$ as $\epsilon\to 0$.
Denote $Y(t)\coloneq W(t)-tW(1)$, so that $W(t)=Y(t)+tW(1)$.
Since $\mathrm{Cov}(Y(t),tW(1))=\mathrm{Cov}(W(t)-tW(1),tW(1))=t\mathrm{Cov}(W(t),W(1
))-t^{2}\mathrm{Var}(W(1),W(1))=0$ and that $Y(t),tW(1)$ are jointly Gaussian, they are independent. Therefore,

$$ $\displaystyle\lim_{\epsilon\to 0}(W(t)\mid\left|W(1)-c\right|<\epsilon)$ $\displaystyle=\lim_{\epsilon\to 0}(Y(t)+tW(1)\mid\left|W(1)-c\right|<\epsilon)$ $\displaystyle=Y(t)+\lim_{\epsilon\to 0}(tW(1)\mid\left|W(1)-c\right|<\epsilon)$ $\displaystyle=W(t)-tW(1)+tc.$ $$

∎

###### Proof.

## Appendix B Additional Results

In this section, we include additional visual and numerical results. In Figure B.3 and Figure B.4, we showcase extended comparisons results with the addition of Control-A-Video (Chen et al., ) and PYoCo (Ge et al., ), along with additional baselines with fixed and interpolated noise using bilinear and nearest interpolating schemes. The corresponding quantitative metrics for both the church and cat scenes are reported in Table 1. In Figure B.2, we use additional examples to showcase the noise contention issue caused by non-injective meshes that applies similarly to our grid-based variant and Chang et al. (), and highlight the robustness of our particle-based variant. In Figure B.1, we show additional results from combining our particle-based, volumetric noise warp with GaussianCube (Zhang et al., ) to facilitate 3D editing.

**Table 1: We show the quality metrics for conditional video generating using SDEdit. The consistency is measured using warp MSE following Chang et al. (), and the realism and faithfulness are measured following Meng et al. ().**
| Video Generation Quality (Church) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | Ours (G) | Ours (P) | HIWYN | PYoCo | CaV | Random | Fixed | Bilinear | Nearest |
| Consistency $\downarrow$ | 9.868e-2 | 1.065e-1 | 1.060e-1 | 1.175e-1 | 1.359e-1 | 1.538e-1 | 1.120e-1 | 8.114e-2 | 1.305e-1 |
| Realism $\downarrow$ | 4.643e-2 | 5.180e-2 | 4.959e-2 | 4.119e-2 | 4.069e-2 | 3.731e-2 | 3.911e-2 | 2.301e-1 | 7.012e-2 |
| Faithfulness $\downarrow$ | 3.872e-2 | 4.309e-2 | 4.377e-2 | 3.764e-2 | 4.169e-2 | 3.976e-2 | 3.264e-2 | 5.623e-2 | 9.321e-2 |
| Video Generation Quality (Cat) |  |  |  |  |  |  |  |  |  |
| Metric | Ours (G) | Ours (P) | HIWYN | PYoCo | CaV | Random | Fixed | Bilinear | Nearest |
| Consistency $\downarrow$ | 6.001e-2 | 5.898e-2 | 5.807e-2 | 6.383e-2 | 4.280e-2 | 1.219e-1 | 3.950e-2 | 3.503e-2 | 1.058e-1 |
| Realism $\downarrow$ | 1.559e-1 | 1.496e-1 | 1.528e-1 | 1.506e-1 | 1.486e-1 | 1.221e-1 | 1.588e-1 | 3.687e-1 | 3.343e-1 |
| Faithfulness $\downarrow$ | 2.039e-2 | 2.064e-2 | 2.022e-2 | 2.023e-2 | 1.817e-2 | 2.077e-2 | 1.972e-2 | 3.809e-2 | 2.201e-1 |

Figure: Figure B.1: Extension of our particle-based variant to perform volumetric noise warping. We show the volume render on the top left, slice views on the top middle, and 3D Gaussians as used in GaussianCube (Zhang et al., ) on the top right. We show that warping the volumetric noise noticeably facilitates temporal consistency over random baseline when we perform 3D editing, which can be observed from the flickering of the window color on the bottom row when random noise is used. We refer to our supplementary video for better visualization of this result.
Refer to caption: x10.png

Figure: Figure B.2: Comparison of grid-based and particle-based variants under non-diffeomorphic deformation maps generated by optical flow (Teed & Deng, ). The orange pixels are the invalid pixels where noise contention occurs. Flow maps are downsampled $10\times$ for better visualization. The test image sequences are borrowed from the dataset by Brox & Malik ().
Refer to caption: x11.png

Figure: Figure B.3: Results generated by all compared methods on the church scene. On the image level, all methods offer similar generation quality except for the interpolation baselines which yield significantly corrupted results. The difference lies in how the details are preserved across frames. Apart from the details of the main tower, the tree on the bottom left also exposes the interesting differences between noise initialization schemes. We refer to our supplementary video for better visualization of these results.
Refer to caption: x12.png

Figure: Figure B.4: Results generated by all compared methods on the cat scene. We observe that HIWYN and our method (both variants) yield similar results, which illustrates the appeal of our particle variant due to its efficiency and simplicity. We refer to our supplementary video for better visualization of these results.
Refer to caption: x13.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 4
questions: See also weaknesses above.

1) The experimental results in the website of the How I Warped Your Noise paper (https://warpyournoise.github.io/), seem of much higher consistency compared to the ones provided in the zip file by the authors. Could the authors comment on this? Were there any reproducibility issues?

2) This [paper](https://arxiv.org/pdf/2410.16152) claims that "white noise is not compatible with the idea of using a generative model to interpolate deformed functions". Could the authors comment on this and the preservation of the Gaussian prior under different transformations?

3) Could the authors investigate if having an equivariant generator improves the experimental results?
soundness: 3
strengths: 1) The paper is very well written and the ideas are presented in a natural way.

2) The method is significantly faster compared to the baseline.

3) The fuzzy variant of the method is very clever: significant computation time is saved and the method is more robust as evaluated in real-world applications.

4) The ideas presented are sound and the Euclidean View offers an alternative way to think about the problem of noise warping that I enjoyed reading about.
summary: The topic of the paper is to develop computationally efficient algorithms to use image diffusion models temporally, i.e. for generating consistent videos or solving video inverse problems. The authors innovate on the noise-warping mechanism and they propose a computationally efficient method to transform the noise between frames that preserves the Gaussian prior distribution. The authors experimentally validate their method for video super-resolution and 3-D problems.
weaknesses: 1) The biggest weakness of this paper is that the experimental results are very weak. I downloaded the supplementary material and looked at the videos; the quality is unsatisfactory. While the proposed method offers more consistency than the existing baselines, it is still very inconsistent. The extent of inconsistency is so high, that I can't think of any applications that would leverage this method. 

2) At the same time, I am not very confident that the considered approach is meaningful. It is true that at the moment there are many powerful image diffusion models and not so many video diffusion models. However, the current level of quality of solving inverse problems with image diffusion models appears to be very low, judging from the results of this paper. I am not sure if this is a fundamental limitation of the approach or whether it will be resolved in the future. But also, as time goes by, more and more powerful video diffusion models are becoming available that can be used as priors for solving temporal inverse problems.

3) I believe that one of the reasons the experimental results are not that strong is the equivariance of the network itself. Even if the input noise transformations are properly handled, maybe the network itself does not provide equivariant predictions which could cause the creation of flickering and inconsistencies. This is argued in this work: https://arxiv.org/pdf/2410.16152. 

4) I also believe that the paper would benefit from more experiments on other inverse problems. Since a pre-trained prior is available, one could experiment with other types of problems such as inpainting or even denoising as the image follows simple geometrical transformations (zoom-in/out, shifts, etc).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Since integral noise warping does not prevent noticeable inconsistency in the details in the frames, is it possible to enhance your method by adding an explicit temporal consistency term between frames?
soundness: 4
strengths: - The new interpretation/formulation of integral noise warping is novel and leads to theoretically backed relation to Brownian bridges, whose Markovian property allows to derive convenient implementation methods.

- The derived methods are useful in practical applications where image-based diffusion models are used for video generation in a training-free manner.

- The paper is well-written.
summary: The paper proposed a different formulation for the "integral noise warping" approach of Chang et al. (2024). The authors prove the relation of this formulation (in infinite resolution) to Brownian bridges, and use it to derive novel implementations for the integral noise warping that do not require upsampling, and thus significantly reduce the computational cost of the method proposed by Chang et al. (2024).
weaknesses: - There is still inconsistency in the details in the frames.  

- The quantitative SDEdit-based video results of the method (and HIWYN) are similar to the competitors.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 2
questions: 1. The symbols are not clearly defined, for example, in the description of the Eulerian Procedure on lines 158-166, $M, m_k, l_k ...$ which can lead to difficulties in reading and understanding.

2. The ultimate computation goal is the same as in HIWYN: $ \tilde I_{i,j}$ , as shown in Figure 1, the goal is to obtain the "area" of the red region $ \tilde A_{i,j}$ . However, from the Eulerian View, it is evident that $ \tilde A_{i,j} $ will be dispersed across each $ A_{i,j}$ ; in the example in Figure 2, this accounts for 39%. $ I_{i,j}$ can be considered a Brownian bridge from 0 to $ I_{i,j} $, and the area of each deformed region can be viewed as "time." Using this, it might be possible to directly sample the 39% value of the Brownian bridge and then gather it with other red parts across various $ A_{i,j} $ regions to obtain the final $ \tilde A_{i,j}$ . I am not sure if my understanding is correct. If this understanding is correct, then the differently colored regions within $A_{i,j}$ should originate from sampling with different conditions (as indicated by equation (1)), meaning that each color belongs to a different distribution. Would it then be valid to interpret this as an addition of values from Brownian bridges?
soundness: 4
strengths: - By using Brownian bridge increments instead of high-resolution upsampling, the proposed method achieves efficient noise manipulation without sacrificing accuracy, greatly enhancing speed and memory efficiency.
- The method maintains temporal consistency of noise in video generation, avoiding flickering or inconsistencies between frames seen in traditional methods, resulting in smoother and more natural video outputs.
- The particle-based noise overlap calculation method is not only faster but also more robust to non-injective mappings, making it suitable for complex, non-linear real-world scenarios and offering high application potential.
summary: The paper presents an advanced noise manipulation method to enhance temporal consistency in video generation using diffusion models. The authors address the computational inefficiency in Chang et al.'s (2024) method, which ensures temporal consistency but requires high-resolution upsampling. By shifting from a Lagrangian to an Eulerian perspective and using Brownian bridge increments instead of high-resolution upsampling, the approach achieves the same precision while significantly improving speed and memory efficiency. The author also introduces two methods for calculating noise overlap areas—grid-based and particle-based. The particle-based approach is faster, more memory-efficient, and more robust to non-injective mappings. Results show that this method not only preserves Gaussian noise properties but also achieves speed improvements by several orders of magnitude, maintaining or even improving temporal consistency in video applications, making it more practical for real-world scenarios.
weaknesses: See Questions.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Not many questions, but I’m curious to know if the authors have any idea what practical use cases could actually benefit from the almost real-time version of noise warping they proposed?
soundness: 4
strengths: 1. While the link between noise warping and brownian bridges was noted in previous work, the insights provided by the Eulerian perspective to leverage this connection are novel and theoretically sound.

2. The proposed algorithms greatly improve the computational efficiency of the problem compared to previous work, which can open the door for more potential applications.

3. The particle-based algorithm tackles another limitation of previous work, which was limited to diffeomorphic warpings. This simpler approach can make such method more accessible.

4. The paper is well-written and clear in its explanations, and the proposed method is well validated.
summary: This paper aims to tackle the computational inefficiency of the noise warping algorithm first presented by Chang et al. [1]. To this end, the authors propose an Eulerian perspective of the warping problem, which allows achieving equivalent results with orders of magnitude speed-up. The algorithm scatters increments of Brownian bridges from the original pixels to the warped ones, effectively removing the need for a high-resolution sample, as it simulates the case of infinite resolution. Additionally, the paper proposes a particle-based approach which is simpler to implement, more robust and even faster, at the cost of minor loss in accuracy in real-world scenarios. To support the practical value of the proposed method, the authors show applications with video translation tasks and 3D editing.

[1] How I Warped Your Noise: a Temporally-correlated noise prior for diffusion models. (Chang et al.)
weaknesses: 1. It is hard to tell from the supplementary video results whether noise warping is actually improving coherency, especially in the 3D case.
2. While the noise warping method is now faster, the authors do not show what practical benefits this speed-up can bring (e.g. what new applications does this open the door to), which slightly hinders the significance of this work.
3. As mentioned by the authors, the method is still limited by the accuracy of the provided flow, and the effects of noise warping in video translation tasks remain based on empirical observations.

Meta-review: This paper aims to develop computationally efficient algorithms to use image diffusion models. An alternative algorithm of the integral noise warping approach by Chang et al. (2024) is presented. It achieves infinite-resolution accuracy while simultaneously reducing the computational cost by orders of magnitude. After the rebuttal, it receives mixed ratings, including one reject, one borderline accept, and two accept. The strength of the paper, including the clear motivation, interesting ideas, extensive experiments, and good results, are well recognized. I agree with them and think the current manuscript meets the requirements of this top conference. Please also address the Reviewer wNg9's concerns in the revised manuscript.

Decision: Accept (Poster)