Edge-preserving noise for diffusion models

## Abstract

Abstract Classical generative diffusion models learn an isotropic Gaussian denoising process, treating all spatial regions uniformly, thus neglecting potentially valuable structural information in the data. Inspired by the long-established work on anisotropic diffusion in image processing, we present a novel edge-preserving diffusion model that is a generalization of denoising diffusion probablistic models (DDPM). In particular, we introduce an edge-aware noise scheduler that varies between edge-preserving and isotropic Gaussian noise. We show that our model’s generative process converges faster to results that more closely match the target distribution. We demonstrate its capability to better learn the low-to-mid frequencies within the dataset, which plays a crucial role in representing shapes and structural information.
Our edge-preserving diffusion process consistently outperforms state-of-the-art baselines in unconditional image generation. It is also more robust for generative tasks guided by a shape-based prior, such as stroke-to-image generation. We present qualitative and quantitative results showing consistent improvements (FID score) of up to 30% for both tasks. We provide source code and supplementary content via the public domain edge-preserving-diffusion.mpi-inf.mpg.de .

## 1 Introduction

Previous work on diffusion models mostly uses isotropic Gaussian noise to transform an unknown data distribution into a known distribution (e.g., normal distribution), which can be
analytically sampled (Song and Ermon, ; Song et al., ; Ho et al., ; Kingma et al., ).
Due to the isotropic nature of the noise, all regions in the data samples $\mathbf{x}_{0}$ are uniformly corrupted, regardless of the underlying structural content, which is typically distributed in a non-isotropic manner. During the backward process, the model is trained to learn an isotropic *denoising* process that ignores this potentially valuable non-isotropic information.
In image processing literature (Elad et al., ), denoising is a well studied topic. Following the work by Perona and Malik () structure-aware guidance has shown remarkable improvements in denoising.
Since generative diffusion models can also be seen as *denoisers*, we ask ourselves: Can we enhance the effectiveness of the generative diffusion process by incorporating awareness of the structural content of the data samples in the underlying dataset?

To explore our question, we introduce a new class of diffusion models that explicitly learn a content-aware noise scheme. We call our noise scheme *edge-preserving noise*, which offers several advantages. First, it allows the backward generative process to converge more quickly to accurate predictions. Second, our edge-preserving model better captures the low-to-mid frequencies in the target dataset, which typically represent shapes and structural information. Consequently, we achieve better results for unconditional image generation. Lastly, our model also demonstrates greater robustness and quality for generative tasks that rely on shape-based priors.

To summarize, we make the following contributions:

- •
We present a novel class of content-aware diffusion models and show how it is a generalization of DDPM (Ho et al., ).
- •
We conduct a frequency analysis to better understand the modeling capabilities of our edge-preserving model.
- •
We run extensive qualitative and quantitative experiments across a variety of datasets to validate the superiority of our model over existing models.
- •
We observed consistent improvements in pixel space diffusion. We found that our model converges faster to more accurate predictions and better learns the low-to-mid frequencies of the target data, resulting in FID score improvements of up to 30% for unconditional image generation and most remarkably a more robust behaviour and better quality on generative tasks with a shape-based prior.

## 2 Related work

Most existing diffusion-based generative models (Sohl-Dickstein et al., ; Song and Ermon, ; Song et al., ; Ho et al., ) corrupt data samples by adding noise with the same variance to all pixels.
These generative models can generate diverse novel content when the noise variance is higher. On the contrary, noise with lower variance is known to preserve the underlying content of the data samples.
Rissanen et al. () proposed to use an inverse heat dissipation model (IHDM), which can be seen as an isotropic Gaussian blurring model. The idea is to isotropically blur the images to corrupt them.
Hoogeboom and Salimans () proposed an improved version of IHDM, where they not only blur but also use isotropic noise to corrupt data samples. The resulting model gives far better quality improvements compared to IHDM.
Recently, Huang et al. () introduced correlated noise for diffusion models (BNDM). They proposed to use *blue noise* which is negatively correlated and showed noticeable improvements both in visual quality and FID scores. While IHDM and BNDM also consider a form of non-isotropic noise, they do not explicitly take into account the structures present in the signal.

Various efforts (Bansal et al., ; Daras et al., ) were made to develop non-isotropic noise models for diffusion processes.
Dockhorn et al. () proposed to use critically-damped Langevin diffusion where the data variable at any time is augmented with an additional ”velocity” variable. Noise is only injected in the velocity variable.
Voleti et al. () performed a limited study on the impact of isotropic vs non-isotropic Gaussian noise for a score-based model. The idea behind non-isotropic Gaussian noise is to use noise with different variance across image pixels.
They use a non-diagonal covariance matrix to generate non-isotropic Gaussian noise, but their sample quality did not improve in comparison to the isotropic case.
Yu et al. () developed this idea further and proposed a Gaussian noise model that adds noise with non-isotropic variance to pixels. The variance is chosen based on how much a pixel or region needs to be edited. They demonstrated a positive impact on editing tasks.

Our definition of anisotropy follows directly from the seminal work by Perona and Malik () on anisotropic diffusion for image filtering. We apply a non-isotropic variance to pixels in an edge-aware manner, meaning that we suppress noise on edges.

## 3 Background

#### Generative diffusion processes.

A generative diffusion model can be described as a parameterized Markov chain where the next (perturbed) state of a data sample $\mathbf{x}_{t+1}$ will only depend on the current state $\mathbf{x}_{t}$ at time $t$. Previous works on generative diffusion models (Song and Ermon, ; Song et al., ; Ho et al., ; Kingma et al., ; Rissanen et al., ; Hoogeboom and Salimans, ) are typically formulated as a Markovian Gaussian process defined within a time interval $t\in[0;T]$ and denoted by the following equation:

$$ $\mathbf{x}_{t}=\gamma_{t}\mathbf{x}_{0}+\sigma_{t}\boldsymbol{\epsilon}_{t}$ (1) $$

here, $\mathbf{x}_{t}$ is the data sample diffused up to time $t$, $\mathbf{x}_{0}$ stands for the original data sample, $\boldsymbol{\epsilon}_{t}$ is a standard normal Gaussian noise, and the signal coefficient $\gamma_{t}$ and noise coefficient $\sigma_{t}$ determine the signal-to-noise ratio (SNR) ($\nicefrac{{\gamma_{t}}}{{\sigma_{t}}}$). The SNR refers to the proportion of signal retained relative to the amount of injected noise. Note that $\gamma_{t}$ and $\sigma_{t}$ are both scalars. Previous work on diffusion models has made several different choices for $\gamma_{t}$ and $\sigma_{t}$ respectively, leading to different variants, each with their own advantages and limitations.
Typically, a noise schedule $\beta_{t}$ is employed to govern the rate at which $\gamma_{t}$ and $\sigma_{t}$ vary over time (Ho et al., ). Let us first define $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha_{t}}=\prod_{i=1}^{t}\alpha_{i}$.
Song and Ermon () then defined their forward diffusion process with $\gamma_{t}=1$ and $\sigma_{t}=\sqrt{1-\bar{\alpha_{t}}}$, leading to a process with exploding variance.
Instead, Ho et al. () chose $\gamma_{t}=\sqrt{\bar{\alpha_{t}}}$ and $\sigma_{t}=\sqrt{1-\bar{\alpha_{t}}}$ which better preserves variance.
Rissanen et al. (), on the other hand, consider a forward process that serves as a low-pass filter to corrupt the information in the data, i.e. blurring in the context of image data.
Their forward heat-dissipation process is formulated as:

$$ $\mathbf{x}_{t}=\mathcal{A}_{t}\mathbf{x}_{0}+\sigma\boldsymbol{\epsilon}_{t}$ (2) $$

where $\mathcal{A}_{t}=VZV^{T}$ is a blurring operator based on the discrete cosine transform $V^{T}$.
Note that in Eq. 2 the noise coefficient $\sigma$ is kept constant, in contrast to previous works following Eq. 1.
This formulation is based on the observation that the solution to the heat dissipation equation:

$$ $\frac{\partial\mathbf{x}}{\partial t}=c(\mathbf{x},t)\Delta\mathbf{x},$ (3) $$

can be formulated as the solution to an eigenvalue problem in the frequency domain.
In Eq. 3, $\Delta\mathbf{x}$ represents the image Laplacian and $c(\mathbf{x},t)$ is a scalar diffusion coefficient, implying isotropic diffusion.
Hoogeboom and Salimans () showed that the forward heat-dissipation process (isotropic blurring) in the spatial domain is equivalent to a non-isotropic diffusion process in the frequency domain. In particular, a different $\gamma_{t}$ is assigned to each discrete frequency component of $V^{T}$.

#### Forward and backward process posteriors.

The isotropic diffusion process formulated in Eq. 1 has the following marginal distribution (see Kingma et al. () for more details):

$$ $q(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}(\gamma\mathbf{x}_{0},\sigma_{t}^{ 2}\boldsymbol{I})$ (4) $$

Moreover, it has the following Markovian transition probabilities:

$$ $q(\mathbf{x}_{t}|\mathbf{x}_{s})=\mathcal{N}(\gamma_{t|s}\mathbf{x}_{s},\sigma _{t|s}^{2}\boldsymbol{I})$ (5) $$

with the forward posterior signal coefficient
$\gamma_{t|s}=\frac{\gamma_{t}}{\gamma_{s}}$
and the forward posterior variance (or square of the noise coefficient)
$\sigma_{t|s}^{2}=\sigma_{t}^{2}-\gamma_{t|s}^{2}\sigma_{s}^{2}$
and $0<s<t<T$.
For a Gaussian diffusion process, given that $q(\mathbf{x}_{s}|\mathbf{x}_{t},\mathbf{x}_{0})\propto q(\mathbf{x}_{t}|
\mathbf{x}_{s})q(\mathbf{x}_{s}|\mathbf{x}_{0})$, one can analytically derive a backward process that is also Gaussian, and has the following marginal distribution:

$$ $q(\mathbf{x}_{s}|\mathbf{x}_{t},\mathbf{x}_{0})=\mathcal{N}(\boldsymbol{\mu}_{ t\rightarrow s},\sigma_{t\rightarrow s}^{2}\boldsymbol{I}).$ (6) $$

The backward posterior variance $\sigma_{t\rightarrow s}^{2}$ has the following form:

$$ $\sigma_{t\rightarrow s}^{2}=\left(\frac{1}{\sigma_{s}^{2}}+\frac{\gamma_{t|s}^ {2}}{\sigma_{t|s}^{2}}\right)^{-1}$ (7) $$

and the backward posterior mean $\boldsymbol{\mu}_{t\rightarrow s}$ is formulated as:

$$ $\boldsymbol{\mu}_{t\rightarrow s}=\sigma_{t\rightarrow s}^{2}\left(\frac{ \gamma_{t|s}}{\sigma_{t|s}^{2}}\mathbf{x}_{t}+\frac{\gamma_{s}}{\sigma_{s}^{2} }\mathbf{x}_{0}\right).$ (8) $$

The analytic formulation of these backward posteriors that model a Gaussian process (time-varying mean and variance) is the theoretical foundation that makes it possible to generate a sample $\mathbf{x}_{0}$ from the data distribution $p_{0}(\mathbf{x})$ at $t=0$, given a fully diffused sample $\boldsymbol{x}_{T}$ from $p_{T}(\mathbf{x})$ at time $t=T$.
With ”fully diffused”, we mean that the sample has converged to a state distributed according to $\mathcal{N}(0,\boldsymbol{I})$.
A practical issue is that Eq. 8 itself depends on the unknown $\mathbf{x}_{0}$, the sample we are trying to generate.
To overcome this, one can instead approximate the analytic reverse process in which $\mathbf{x}_{0}$ is replaced by its approximator $\hat{\mathbf{x}}_{0}$, learned by a deep neural network $f_{\theta}(\boldsymbol{x}_{t},t)$.
The network can learn to directly predict $\mathbf{x}_{0}$ given an $\mathbf{x}_{t}$ (a sample with a level of noise that corresponds to time $t$), but previous work has shown that it is beneficial to instead optimize the network to learn the approximator $\hat{\boldsymbol{\epsilon}}_{t}$. $\hat{\boldsymbol{\epsilon}}_{t}$ predicts the unscaled Gaussian white noise that was injected at time $t$.
$\hat{\mathbf{x}}_{0}$ can then be obtained via Eq. 9, which follows from Eq. 1.

$$ $\hat{\mathbf{x}}_{0}=\frac{1}{\gamma_{t}}\mathbf{x}_{t}-\frac{\sigma_{t}}{ \gamma_{t}}\hat{\boldsymbol{\epsilon}}_{t}$ (9) $$

#### Edge-preserved filtering in image processing.

A classic and effective technique for denoising in the field of image processing is edge-preserved filtering via anisotropic diffusion (Weickert et al., ). Perona and Malik () showed that a Gaussian scale-space (a family of images obtained by convolving the original image $\mathbf{x}_{0}$ with Gaussian kernels of varying scales) can also be expressed in terms of a diffusion process. Long ago, Fourier () already found that the solution to the heat equation (Eq. 3) has a special form, namely that of a Gaussian convolution with the original signal $\mathbf{x}_{0}$. This observation has interesting consequences, namely that the blurred version of a digital signal can be computed by integrating the discretized Laplacian of that signal over time. However, if the goal is to remove noise from an image, a disadvantage of a standard isotropic blurring process (such as described by Eq. 3) is that the relevant structural information in the image also gets distorted. To overcome this problem, Perona and Malik () propose an anisotropic diffusion process of the form:

$$ $\mathbf{x}_{t}=\mathbf{x}_{0}+\int_{0}^{t}\mathbf{c}(\mathbf{x}_{s},s)\Delta \mathbf{x}_{s}\,ds$ (10) $$

where the diffusion coefficient $\mathbf{c}(\mathbf{x}_{s},s)$ is this time a tensor (unlike in Eq. 3) and takes the following form:

$$ $\mathbf{c}(\mathbf{x},t)=\frac{1}{\sqrt{1+\frac{||\nabla\mathbf{x}_{t}||}{ \lambda}}}$ (11) $$

where $||\nabla\mathbf{x}||$ is the gradient magnitude image, and $\lambda$ is the edge sensitivity. Intuitively, in the regions of the image where the gradient response is high (on edges), the diffusion coefficient will be smaller, and therefore the signal gets less distorted there. The edge sensitivity $\lambda$ determines how sensitive the diffusion coefficient is to the image gradient response.

## 4 Edge-preserving noise

Inspired by  Perona and Malik (), we aim to design a linear diffusion process that incorporates edge-preserving noise. To obtain a content-aware linear diffusion process, we apply the idea of edge-preserved filtering (anisotropic diffusion) to the noise term of Eq. 1. More specifically, we define an edge-preserving forward process as follows:

$$ $\mathbf{x}_{t}=\sqrt{\bar{\alpha_{t}}}\mathbf{x}_{0}+\frac{\sqrt{1-\bar{\alpha _{t}}}}{\sqrt{1+\frac{||\nabla\mathbf{x}_{0}||}{\lambda(t)}}}\boldsymbol{ \epsilon}_{t}$ (12) $$

Observe that formally, our forward process looks very similar to the isotropic forward process introduced in Eq. 1: we made the same choice $\gamma_{t}=\sqrt{\bar{\alpha_{t}}}$ for the signal coefficient as DDPM (Ho et al., ), and only modified the noise coefficient $\sigma_{t}$. In particular, we multiply DDPM’s choice for $\sigma_{t}=\sqrt{1-\bar{\alpha_{t}}}$ by a non-isotropic diffusion coefficient $\mathbf{c}(\mathbf{x}_{0},t)$ defined in Eq. 11. This results in the noise coefficient for our process becoming a tensor $\boldsymbol{\sigma}_{t}$ instead of a scalar $\sigma_{t}$. Intuitively, we suppress noise on the edges and leave noise unchanged in more uniform image regions.
In our formulation, we also consider $\lambda$ to be time-varying (more details in section Section 4.2).
Furthermore, it is important to note that we do not consider the original anisotropic diffusion as presented in Eq. 10. Instead, our diffusion coefficient only takes into account the gradient magnitude of $\mathbf{x}_{0}$. In the original formulation, the diffusion coefficient $\mathbf{c}(\mathbf{x},t)$ is time-dependent (it depends on the state of the data sample $\mathbf{x}_{t}$ at each point in time) and therefore the resulting forward process would no longer be linear.

### 4.1 Forward hybrid noise scheme

The forward *edge-preserving* (anisotropic) process described in Eq. 12 in its pure form is not very meaningful in our setup. This is because if the edges are preserved all the way up to time $t=T$, we end up with a rather complex distribution $p_{T}(x)$ that we cannot analytically take samples from. Instead, we would like to end up with a well-known *prior* distribution at time $t=T$, such as the standard normal distribution. To achieve this, we instead consider the following hybrid forward process:

$$ $\mathbf{x}_{t}=\sqrt{\bar{\alpha_{t}}}\mathbf{x}_{0}+\frac{\sqrt{1-\bar{\alpha _{t}}}}{(1-\tau(t))\sqrt{1+\frac{||\nabla\mathbf{x}_{0}||}{\lambda(t)}}+\tau(t )}\boldsymbol{\epsilon}_{t}$ (13) $$

The function $\tau(t)$ now appearing in the denominator of the diffusion coefficient is the transition function. When $\tau(t)<1$, we obtain edge-preserving noise (the edge-preservation is stronger when $\tau(t)\approx 0$). The turning point where $\tau(t)=1$ is called the transition point $t_{\Phi}$. At the transition point, we switch over to isotropic noise with scalar noise coefficient $\sigma_{t}=\sqrt{1-\bar{\alpha_{t}}}$. This approach allows us to flexibly design noise schedulers that start off with edge-preserving noise and towards the end of the forward process fall back to an isotropic diffusion coefficient. Practically, one can choose any function for $\tau(t)$, as long as it maps to $[0;1]$ and $\tau(t)=1$ for $t$ in proximity to $T$. We performed an ablation for different transition functions in Section 5.2. We would like to draw attention to how our model is a generalization of DDPM. If we set the transition function to be a constant $\tau(t)=1$, we end up with exactly the forward process used in DDPM.
Choosing any other non-constant function for $\tau(t)$ leads to a hybrid diffusion process that consists of a mix of an edge-preserving stage and an isotropic stage (at $\tau(t)=1$).

### 4.2 Time-varying edge sensitivity λ ⁢ ( t ) 𝜆 𝑡 \lambda(t) italic_λ ( italic_t )

The edge sensitivity parameter $\lambda$ controls the granularity of the content preservation on image edges. By setting $\lambda$ very low, all levels of edges and fine details are preserved. The higher we increase $\lambda$, the less details will be preserved.
By choosing a very high $\lambda$ (e.g. $\lambda>1e^{-1}$), the edge-preserving stage of the diffusion process almost resembles an isotropic process because the noise suppression on the edges is unnoticeable. We study this impact in detail in the ablation study in Section 5.2.

[Uncaptioned image]: x2.png

We found that choosing constant values for $\lambda$ has negative impact on sample quality. Selecting a $\lambda$-value that is too low results in unrealistic, ”cartoonish” outputs, while a $\lambda$-value that is too high diminishes the effectiveness of the edge-preserving diffusion model, making it nearly indistinguishable from an isotropic model.

To overcome this, we instead consider a time-varying edge sensitivity $\lambda(t)$. We set an interval $[\lambda_{min};\lambda_{max}]$ that bounds the possible values for the time-varying edge sensitivity. The function that governs $\lambda(t)$ within this interval can in theory again be chosen freely. We have so far experimented with a linear function and a sigmoid function. We experienced that a linear function for $\lambda(t)$ resulted in higher sample quality and therefore used this function for our experiments. Additionally, we have attempted to optimize the interval $[\lambda_{min};\lambda_{max}]$, but this led to unstable behaviour.

### 4.3 Backward process posteriors and training

Given our forward hybrid diffusion process introduced in Section 4.1, we can derive the actual formulations for the posterior mean $\boldsymbol{\mu}_{t\rightarrow s}$ and variance $\boldsymbol{\sigma}_{t\rightarrow s}^{2}$ for the corresponding backward process. To do this, we simply fill in Eq. 7 and Eq. 8 with our choices for the signal coefficient $\gamma_{t}$ and variance $\boldsymbol{\sigma}_{t}^{2}$. Recall that we chose $\boldsymbol{\sigma}_{t}^{2}$ to be a tensor, which is why the backward posterior variance $\boldsymbol{\sigma}_{t\rightarrow s}^{2}$ is again a tensor, contrary to isotropic diffusion processes considered in previous works. Regardless, we can use the same equations and the algebra still works.

We first introduce an auxiliary variable $\boldsymbol{\sigma}^{2}(t)$, which represents the variance of our forward process at a given time $t$. This is simply the square of our choice for the noise coefficient $\boldsymbol{\sigma}_{t}$ formulated in  Eq. 13:

$$ $\boldsymbol{\sigma}^{2}(t)=\frac{1-\bar{\alpha_{t}}}{(1-\tau(t))^{2}\left(1+ \frac{||\nabla\mathbf{x}_{0}||}{\lambda(t)}\right)+2\left((1-\tau(t))\sqrt{1+ \frac{||\nabla\mathbf{x}_{0}||}{\lambda(t)}}\tau(t)\right)+\tau(t)^{2}}$ (14) $$

Here $\bar{\alpha_{t}}$ has the same meaning as earlier described in Section 3. We now have the backward posterior variance $\boldsymbol{\sigma}_{t\rightarrow s}^{2}$:

$$ $\boldsymbol{\sigma}_{t\rightarrow s}^{2}=\left(\frac{1}{\boldsymbol{\sigma}^{2 }(t)}+\frac{\frac{\bar{\alpha_{t}}}{\bar{\alpha_{s}}}}{\boldsymbol{\sigma}^{2} (t)-\frac{\bar{\alpha_{t}}}{\bar{\alpha_{s}}}\boldsymbol{\sigma}^{2}(s)}\right )^{-1}$ (15) $$

and the backward posterior mean $\boldsymbol{\mu}_{t\rightarrow s}$:

$$ $\boldsymbol{\mu}_{t\rightarrow s}=\sigma_{t\rightarrow s}^{2}\left(\frac{\frac {\sqrt{\bar{\alpha_{t}}}}{\sqrt{\bar{\alpha_{s}}}}}{\boldsymbol{\sigma}^{2}(t) -\frac{\bar{\alpha_{t}}}{\bar{\alpha_{s}}}\boldsymbol{\sigma}^{2}(s)}\mathbf{x }_{t}+\frac{\sqrt{\bar{\alpha_{s}}}}{\boldsymbol{\sigma}^{2}(s)}\mathbf{x}_{0}\right)$ (16) $$

Given Eq. 15 and Eq. 16, the only unknown preventing us from simulating the Gaussian backward process is $\mathbf{x}_{0}$.

Figure: Figure 2: We visually compare the impact of our anisotropic edge-preserving noise on the generation (reverse) process. In each column, we show predictions $\hat{\mathbf{x}}_{0}$ at selected time steps. Our method converges significantly faster to a sharper and less noisy image than its isotropic counterpart. This is evident by the earlier emergence (from $t=400$) of structural details like the pattern on the cat’s head, eyes, and whiskers with our approach.
Refer to caption: extracted/5953683/figures/pred_x0s_comparison/ddpm/0.png

Note that $\mathbf{x}_{0}$ in our case depends on a non-isotropic noise.
Therefore, we cannot just use an isotropic approximator $\hat{\boldsymbol{\epsilon}}_{t}$ for the isotropic noise $\boldsymbol{\epsilon}_{t}$ to predict $\hat{\mathbf{x}}_{0}$ via Eq. 9.
Instead, we need a model that can predict the non-isotropic noise $\boldsymbol{\sigma}_{t}\boldsymbol{\epsilon}_{t}$ .
We introduce the loss function that trains such an approximator:

$$ $\mathcal{L}=||f_{\theta}(\boldsymbol{x}_{t},t)-\boldsymbol{\sigma}_{t} \boldsymbol{\epsilon}_{t}||^{2}.$ (17) $$

It is very similar to the loss function used in DDPM, with the difference that our model explicitly learns to predict the non-isotropic ($\boldsymbol{\sigma}_{t}\boldsymbol{\epsilon}_{t}$) edge-preserving noise. $f_{\theta}(\boldsymbol{x}_{t},t)$ stands for the time-conditioned U-Net used to approximate the time-varying noise function. The visual difference between the backward process of an isotropic diffusion model (DDPM) and ours is shown in Fig. 2. Our formulation introduces a negligible overhead. The only additional computation that needs to be performed is the image gradient $||\nabla\mathbf{x}||$, which can very efficiently be done on modern GPUs. We have not noticed any significant difference in training times between vanilla DDPM and our method.

#### Discussion.

A valid observation to make is that given our hybrid forward process with two distinct stages, the edges are preserved longer, but still lost in the end. How does longer preservation of edges help the generative process? First thing to note is that the longer preservation of edges by itself does not have any impact, if we still let the model predict isotropic noise. Secondly, by modifying the forward process to be an edge-preserving one, the backward posterior formulation will also change and will rely on a non-isotropic variance, as discussed in this subsection. It is the combination of edge-preserving noise, together with our structure-aware loss function that makes the model work. Furthermore, our frequency analysis (Section 5.1) has quantitatively shown that our decoupling approach is beneficial to learning the low-to-mid frequencies of the target dataset. This is consistent with recent work on wavelet-based diffusion models (Huang et al., ), that demonstrates it is advantageous to learn low-frequency content separately from high-frequency content in the wavelet domain, using two distinct modules. Instead, we use two distinct stages, one that focuses on lower-frequency primary structural content (edge-preserving stage), and one that focuses on fine-grained high-frequency details (isotropic stage).

## 5 Experiments

#### Implementation details

Figure: Figure 3: We compare unconditionally generated samples for IHDM, BNDM, DDPM with our model. Ours perform consistently better both qualitatively and quantitatively. Corresponding FID scores can be found in Table 1. More results are presented in the appendix.
Refer to caption: extracted/5953683/figures/comparison/cat128/ihdm.png

We compare our method against three baselines, namely DDPM (Ho et al., ), IHDM (Rissanen et al., ) and BNDM (Huang et al., ). The motivation for comparing with the latter two works is that they also consider a non-isotropic form of noise.

We perform experiments on two settings: pixel-space diffusion following the setting of  Ho et al. (); Rissanen et al. () and latent-space diffusion following (Rombach et al., ) noted as LDM in Table 1, where the diffusion process can be driven by any method but runs in the latent space.
We use the following datasets: CelebA ($128^{2}$, 30,000 training images) (Lee et al., ), AFHQ-Cat ($128^{2}$, 5,153 training images) (Choi et al., ), Human-Sketch ($128^{2}$, 20,000 training images)  (Eitz et al., ) (see Appendix) and LSUN-Church ($128^{2}$, 126,227 training images) (Yu et al., ) for pixel-space diffusion.
For high-resolution image generation with latent-space diffusion (Rombach et al., ), we use on CelebA ($256^{2}$), AFHQ-Cat ($512^{2}$).

We used a batch size of 64 for all experiments in image space, and a batch size of 128 for all experiments in latent space.
We trained AFHQ-Cat ($128^{2}$) for 1000 epochs, AFHQ-Cat ($512^{2}$) (latent diffusion) for 1750 epochs, CelebA($128^{2}$) for 475 epochs, CelebA($256^{2}$) (latent diffusion) for 1000 epochs and LSUN-Church($128^{2}$) for 90 epochs for our method and all baselines we compare to.
Our framework is implemented in Pytorch (Paszke et al., ).
For the network architecture we use a 2D U-Net from Rissanen et al. ().
We use T = 500 discrete time steps for both training and inference, except for AFHQ-Cat128, where we used T = 750.
To optimize the network parameters, we use Adam optimizer (Kingma and Ba, ) with learning rate $1e^{-4}$ for latent-space diffusion models and $2e^{-5}$ for pixel-space diffusion models.
We trained all datasets on 2x NVIDIA Tesla A40.

For our final results in image space, we used a linear scheme for $\lambda(t)$ that linearly interpolates between $\lambda_{min}=1e^{-4}$ and $\lambda_{max}=1e^{-1}$. We used a transition point $t_{\Phi}=0.5$ and a linear transition function $\tau(t)$. For latent diffusion, we used $\lambda_{min}=1e^{-5}$ and $\lambda_{max}=1e^{-1}$, with $t_{\Phi}=0.5$ and a linear $\tau(t)$.

To evaluate the quality of generated samples, we consider FID (Heusel et al., ).
using the implementation from Stein et al. (), with Inception-v3 network (Szegedy et al., ) as backbone.
We generate 30k images to compute FID scores for unconditional generation for all datasets.

#### Unconditional image generation

We show unconditional image generation results in Fig. 3 and  Appendix A. The corresponding FID metrics are listed in Table 1.
We observe improvements w.r.t.

**Table 1: Quantitative FID score comparisons among IHDM (Rissanen et al., ), DDPM (Ho et al., ), BNDM (Huang et al., ) and our method across different datasets.**
| FID ($\downarrow$) | CelebA($128^{2}$) | LSUN-Church($128^{2}$) | AFHQ-Cat($128^{2}$) |
| --- | --- | --- | --- |
| IHDM | 89.67 | 119.34 | 53.86 |
| DDPM | 28.17 | 31.00 | 17.60 |
| BNDM | 26.35 | 29.86 | 14.54 |
| Ours | 26.15 | 23.17 | 13.06 |

all baselines both visually and quantitatively. While the visual improvement over DDPM is subtle,
our model generally demonstrated greater robustness to artifacts.
We attribute these improvements to the explicit training of our model on predicting the non-isotropic noise associated with the edges in the dataset.
We also performed comparisons in the latent space, which are listed in Table 2 in Appendix A.
For latent space diffusion (CelebA($256^{2}$) and AFHQ-Cat($512^{2}$)), although our model is slightly outperformed on the FID metric, the visual quality of our samples is often comparable, and at times even superior (see Fig. 12 and Fig. 13 in Appendix A).
This highlights the known limitations of FID, as it doesn’t always reliably capture visual quality (Liu et al., ).
In particular we would also like to draw attention to Fig. 4. The Human-Sketch ($128^{2}$) dataset (Eitz et al., ) is very uncommon to use for evaluation in the diffusion community, however we found it interesting given its content is entirely composed of edges. Results for additional baselines are shown in Appendix A.

Figure: Figure 4: Unconditional samples for the Human Sketch ($128^{2}$) dataset (Eitz et al., ). All models were trained for an equal amount of 575 epochs. Our edge-aware model outperforms all models including DDPM (see Fig. 6).
Refer to caption: extracted/5953683/figures/human_sketch_dataset/ddpm/1.png

Figure: Figure 5: Left: Various diffusion models applied to the SDEdit framework (Meng et al., ) are shown. The leftmost column displays the stroke-based guide, with the other three columns showing the model outputs. Overall our model shows sharper details with less distortions compared to other models, leading to a better visual and quantitative performance. The corresponding FID scores are shown in the top right column. Right: Our model also effectively uses human-drawn paintings as shape guides, with particularly precise adherence to details, such as the orange patches on the cat’s fur, unlike DDPM (middle column).
Refer to caption: extracted/5953683/figures/sde_edit_comparison/celeba/stroke_painting_from_image.png

#### Stroke-guided image generation (SDEdit)

Motivated by the hope that our model would better adhere to the guidance provided by shape-based priors such as sketches, we applied our edge-preserving diffusion model to the SDEdit framework  (Meng et al., ) for sketch-guided image generation. More specifically, we converted a set of 1000 original images from the training set into stroke painting images using k-means clustering. To generate samples, we then use the stroke paintings as inputs to the SDEdit framework, for different diffusion models. We used a hijack point of $0.55T$, meaning that we run 55% of the forward process to get a noisy stroke painting, which is then denoised to obtain the sample. We computed the FID score between the set of original images and the set of generated samples, to measure which method is able to make better reconstructions given a shape-based prior. A comparison is shown in Fig. 5. We observed that our edge-preserving model better adheres to the guiding prior and overall
behaves more robustly on this task. It suffers less from artifacts that are present for the other methods, leading to a significant improvement in performance both qualitatively and quantitatively.

[Uncaptioned image]: x3.png

### 5.1 Frequency analysis of training performance

To better understand our model’s capacity of modeling the target distribution, we conducted an analysis on its training performance for different frequency bands. Our setup is as follows, we create 5 versions of the AFHQ-Cat128 dataset, each with a different cutoff frequency.
This corresponds to convoluting each image in the dataset with a Gaussian kernel of a specific standard deviation $\sigma$, representing a frequency band. For each frequency band, we then trained our model for a fixed amount of 10000 training iterations.
We place a model checkpoint at every 1000 iterations, so we can also investigate the evolution of the performance over this training time. We measure the performance by computing the FID score between 1000 generated samples (for that specific checkpoint) and the original dataset of the corresponding frequency band.
A visualization of the analyzed results is presented in the inline figure on the right.
We found that our model is able to learn the low-to-mid frequencies of the dataset significantly better than the isotropic model (DDPM). The figure shows the evolution of FID score over the first 10,000 training iterations per frequency band (larger $\sigma$ values correspond to lower frequency bands). $a)$ and $b)$ show performance in terms of FID score of DDPM and our model, respectively. $c)$ shows their difference (positive favors our method). $d)$ visualizes the information in 2D for a more accurate comparison. Our model significantly outperforms in low-to-mid frequency bands (lower FID is better).

### 5.2 Ablation study

#### Impact of transition function τ ⁢ ( t ) 𝜏 𝑡 \tau(t) italic_τ ( italic_t ) .

We have experimented with three different choices for the transition function $\tau(t)$: linear, cosine and sigmoid. While cosine and sigmoid show similar performance, we found that having a smooth linear transition function significantly improves the performance of the model.
A qualititative and quantitative comparison between the choices is presented in the inline figure below.

[Uncaptioned image]: x4.png

#### Impact of transition points t Φ subscript 𝑡 Φ t_{\Phi} italic_t start_POSTSUBSCRIPT roman_Φ end_POSTSUBSCRIPT .

We have investigated the impact of the transition point $t_{\Phi}$ on our method’s performance by considering 3 different diffusion schemes: 25% edge-preserving - 75% isotropic, 50% isotropic - 50% edge-preserving and 75% edge-preserving - 25% isotropic. A visual example for AFHQ-Cat ($128^{2}$) is presented in the inline figure on the right. We have experienced that there are limits to how far the transition point can be placed without sacrificing sample quality. Visually, we observe that the further the transition point is placed, the less details the model generates. The core shapes however stay intact. This is illustrated well by Fig. 7 in Appendix A. For the datasets we tested on, we found that the 50%-50% diffusion scheme works best in terms of FID metric and visual sharpness. This again becomes apparent in  Fig. 7: although the samples for $t_{\Phi}=0.25$ contain slightly more details, the samples for $t_{\Phi}=0.5$ are significantly sharper.

#### Impact of edge sensitivity λ ⁢ ( t ) 𝜆 𝑡 \lambda(t) italic_λ ( italic_t ) .

As shown in the above inline figure, lower constant $\lambda(t)$ values lead to less detailed, more flat, ”water-painting-style” results. The intuition behind this is that a lower $\lambda(t)$ corresponds to stronger edge-preserving noise and our model is explicitly trained to accordingly better learn the core structural shapes instead of the high-frequency details that we typically find in interior regions. Our time-varying choice for $\lambda(t)$ works better than other settings in our experiments, by effectively balancing the preservation of structural information across different granularities.

## 6 Conclusion

We introduced a new class of edge-preserving generative diffusion models that extend DDPMs with negligible overhead. In practice, we didn’t notice any increase in training/inference time. Our linear diffusion process has two stages: edge-preserving and isotropic. The edge-preserving stage maintains core shapes, while the isotropic stage fills in details. This decoupled approach captures low-to-mid frequencies better and converges to sharper predictions faster. It improves performance on both unconditional and shape-guided tasks, outperforming several state-of-the-art models.
For future work it would be interesting to extend our non-isotropic framework to the temporal dimension for video generation to improve time-consistency of important image features.

#### Broader impact.

We recognize that releasing generative models can open doors for malevolent actors to exploit and make misuse of them. However, by being transparent about the workings of our method and releasing the source code, we hope to support the research communities that are working on methods to better detect machine-generated content and warn or protect against it.

## Appendix A Appendix

In the appendix we show additional results and ablations.

Table 2 shows quantitative FID comparisons using latent diffusion [Rombach et al., ] models on all the baselines.

Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 show more generated samples and comparisons between IHDM, DDPM on all previously introduced datasets.
In  Fig. 6 we show samples for the Human-Sketch $(128^{2})$ data set specifically. This dataset was of particular interest to us, given
the images only consist of high-frequency, edge content. Although we observed that this data is
remarkably challenging for all methods, our model is able to consistently deliver visually better
results. Note that although we report FID scores for this data set, they are very inconsistent with the visual quality of the samples. This is likely due to the Inception-v3 backbone being designed for continuous image data, leading to highly unstable results when applied to high-frequency binary data.

Figure 7 shows an additional visualization of the impact $t_{\Phi}$ for the LSUN-Church ($128^{2}$) dataset. $t_{\Phi}=0.5$ works best in terms of FID metric, consistent to the results shown in Section 5.2.

Figure: Figure 6: Generated unconditional samples for the Human Sketch ($128^{2}$) dataset [Eitz et al., ]. All models were trained for an equal amount of 575 epochs. Note that the FID scores are inconsistent with visual quality. The cause for this is the Inception-v3 backbone, which is designed for continuous image data, leading to highly unstable results when applied to high-frequency binary data like hand-drawn sketches.
Refer to caption: extracted/5953683/figures/human_sketch_dataset/ihdm/1.png

Figure: Figure 7: Impact of location of transition point $t_{\Phi}$ on sample quality, shown for the LSUN-Church ($128^{2}$) dataset. If we place $t_{\Phi}$ too far, the model happens to learn only the lowest frequencies and generates no details at all. Placing it too early leads to results that are less sharp. We found that by placing $t_{\Phi}$ at 50%, we strike a good balance between the two, leading to better quantitative and qualitative results.
Refer to caption: extracted/5953683/figures/appendix_trans_pt_ablation/0pt25.png

**Table 2: Quantitative FID score comparisons on latent diffusion models [Rombach et al., ] among IHDM [Rissanen et al., ], DDPM [Ho et al., ], BNDM [Huang et al., ] and our method.**
| FID ($\downarrow$) | CelebA($256^{2}$, latent) | AFHQ-Cat($512^{2}$, latent) |
| --- | --- | --- |
| IHDM | 88.12 | 28.09 |
| DDPM | 7.87 | 22.86 |
| BNDM | 10.93 | 13.62 |
| Ours | 13.89 | 18.91 |

Figure: Figure 8: More samples for our model and other baselines applied to SDEdit [Meng et al., ]. Note how our model is able to generate sharper results that suffer less from artifacts. Although BNDM can generate satisfactory results in certain cases (e.g., cat and church), it often deviates from the stroke painting guide, potentially producing outcomes that differ significantly from the user’s original intent. In contrast, our method closely follows the stroke painting guide, accurately preserving both shape and color.
Refer to caption: extracted/5953683/figures/sde_edit_comparison/celeba/extras/painting_458.png

Figure: Figure 9: More unconditional samples for IHDM, DDPM and our method on the AFHQ-Cat ($128^{2}$) dataset. Although the difference between DDPM and our method is subtle, we consistently found that our approach captures geometric details more effectively (e.g., whiskers) and experiences fewer blurry artifacts (e.g., right sample in row 3, DDPM vs. Ours).
Refer to caption: extracted/5953683/figures/cat_res128/ihdm.png

Figure: Figure 10: More unconditional samples for IHDM, BNDM, DDPM and our method on the CelebA ($128^{2}$) dataset. While BNDM is only slightly outperformed by our model in terms of FID metric, its samples look noticeably different in terms of colors. We attribute this difference to the fact that BNDM simulates an ODE, where we in contrast simulate an SDE, possibly causing both methods to sample a different part of the manifold. In terms of visual quality the BNDM samples also show more artifacts, but it is known from previous work that FID score does not always well reflect percepted quality [Liu et al., ].
Refer to caption: extracted/5953683/figures/celeba_res128/ihdm.png

Figure: Figure 11: More unconditional samples for IHDM, BNDM, DDPM and our method on the LSUN-Church ($128^{2}$) dataset. lthough our results appear similar to DDPM’s, our method more effectively captures the geometric details of buildings and exhibits fewer artifacts, such as blurry regions, compared to DDPM.
Refer to caption: extracted/5953683/figures/church_res_128/ihdm.png

Figure: Figure 12: More unconditional samples for IHDM, DDPM and our method on the AFHQ-Cat ($512^{2}$, LDM) dataset. All samples are generated via diffusion in latent space. Note that despite the deficit in FID score, our method is able to produce results of very similar perceptual quality.
Refer to caption: extracted/5953683/figures/ldm_cat_res512/ihdm_cat_512_grid_appendix.png

Figure: Figure 13: More unconditional samples for IHDM, DDPM and our method on the CelebA ($256^{2}$, LDM) dataset. All samples are generated via diffusion in latent space. Although our method is slightly outperformed in terms of the FID metric, the visual quality of our samples is highly comparable to the baselines, and in some cases, even superior (e.g., third row of DDPM vs. Ours).
Refer to caption: extracted/5953683/figures/ldm_celeba_res256/ihdm_celeba_256_grid_appendix.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: None
soundness: 3
strengths: 1. The paper provides a theoretical foundation for the proposed model, explaining how it generalizes DDPM and improves frequency learning.

2. The model's ability to better preserve structural information could have practical benefits in various image generation and editing applications.

3. The paper presents a creative method for improving diffusion models by introducing edge-preserving noise, addressing a challenge in generative modeling.

4. The method shows substantial improvements in both qualitative and quantitative metrics, particularly in reducing FID scores and enhancing image generation quality.
summary: The paper introduces a novel approach to generative diffusion models by incorporating edge-preserving noise, which is a generalization of denoising diffusion probabilistic models (DDPM). The authors propose an edge-aware noise scheduler that alternates between edge-preserving and isotropic Gaussian noise, aiming to better capture the structural information in data. The model demonstrates faster convergence to the target distribution and improved learning of low-to-mid frequencies, which are crucial for shape and structural representation.
weaknesses: 1. The paper does not discuss the computational cost of the proposed method compared to existing approaches, which is an important consideration for practical applications. Suggesting comparisons with existing methods on these metrics could provide valuable context. For instance, a table comparing memory usage and speed (inference and training) across baseline models could highlight this model's practical viability more clearly.

2. While the paper demonstrates improvements on a specific dataset, it is unclear how well the model generalizes to other types of models or tasks. Testing on tasks that emphasize diverse structural features, such as denoising or inpainting, could provide insight into how well the model maintains performance across broader applications.

3. The current evaluation uses limited metrics to assess the effectiveness of the proposed method. Including newer metrics like CLIPScore, etc. could provide a more comprehensive view of performance. Specifically, CLIPScore could offer insights into the model's ability to generate semantically consistent images. Given the model's edge-preserving focus, metrics that capture detail preservation and structure alignment might also be beneficial.

4. While the paper compares the proposed method with existing baselines, it could benefit from a more detailed comparative analysis with other state-of-the-art methods, especially those that also aim to preserve structural information in images.

5. The performance of the proposed model may be highly sensitive to the choice of hyperparameters, such as the transition function and edge sensitivity. The paper could provide **more insight** into how these parameters affect the model's performance and **how to select** them effectively.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the Weakness section.
soundness: 3
strengths: This paper proposes an interesting approach to improve the generation quality of diffusion models.
The experiments are extensive, and the visualizations are clear, helping readers understand the improved generation performance.
summary: This paper introduces a novel edge-preserving diffusion model that generalizes denoising diffusion probabilistic models (DDPMs). By incorporating an edge-aware noise scheduler, the model retains structural information in images, converges more rapidly to the target distribution, and better captures low-to-mid frequency components within datasets, which are crucial for representing shapes and structural details.
weaknesses: 1. Does edge-aware noise affect generation diversity?
2. In the "Related Work" section, you initially describe IHDM as an isotropic Gaussian blurring model, yet later refer to it and BNDM as incorporating a form of non-isotropic noise. This creates a seeming contradiction that would benefit from further clarification to ensure the logical coherence of your manuscript.
3. While the paper makes a commendable effort to integrate the anisotropic diffusion concept from Perona and Malik (1990) into the diffusion model framework, the innovation appears to be primarily an application of existing theory rather than a novel theoretical contribution. The incorporation of edge-preserving noise as a new term in the diffusion process, while potentially enhancing the model's performance in certain aspects, does not fundamentally alter the underlying principles of generative diffusion models.
4. The discussion surrounding Eq. (2) and Eq. (3) in the paper is somewhat perplexing due to its circuitous approach. The authors explore the relationship between diffusion models and heat conduction, as well as the limitations of isotropic noise, before returning to the contributions of Perona and Malik (1990) in Eq. (10). A more effective strategy would be to engage with Perona and Malik's work on edge-preserving diffusion earlier in the discussion. This would create a natural transition to Eq. (12), which introduces the paper's core innovation—a novel noise term.
5. In the "Background" section, when discussing the "Forward and backward process posteriors," you utilize the formulation presented in Eq. (1), which is consistent with the isotropic Gaussian process. However, you also reference Eq. (2), which is derived from the heat dissipation process and represents a different formulation. My concern is that the use of both Eq. (1) and Eq. (2) in this context may create confusion, especially since your novel edge-preserving noise scheme seems to be more aligned with the principles of Eq. (1), as inferred from the derivations in Section 4.2. It appears that your improved noise term is inspired by the anisotropic diffusion work of Perona and Malik, which is more closely related to Eq. (2). Yet, the parameter calculations and posterior calculations in your model follow a form similar to Eq. (1). If this is the case, then the inclusion of Eq. (2) in the introduction might be seen as unnecessary, and it could potentially detract from the clarity of your paper. I would recommend either providing a more explicit connection between your noise model and Eq. (2), or if Eq. (2) is not central to your approach, consider streamlining the presentation to focus on Eq. (1) and its relevance to your edge-preserving noise scheme. This would help readers to better understand the relationship between your work and the existing literature and clarify why both equations are presented if they are not both integral to your model.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * It is not clear whether NLLs can be approximated easily with this approach, potentially limiting the applications. Please clarify or demonstrate this.
* The time-varying edge sensitivity $\lambda(t)$ seems arbitrary in section 5, and is currently dependent on Ho et al.,'s schedule, tying this down to similar variants. 
   * Can the main theory be better abstracted and demonstrated in different diffusion frameworks?
   * Is the approach suited for Flow Matching?
* Implementing this within a modern diffusion framework/architecture that efficiently supports high-resolution datasets would be beneficial to quantitatively evaluate improvements.
   * How does this scale with FFHQ $1024^2$ or ImageNet-$128$?
* Please clarify how $|| \nabla_{\mathbf{x}_0}||$ is implemented. Is this approximated with a Sobel filter?
soundness: 2
strengths: The idea is intuitively simple and straightforward to incorporate into generative imaging applications. It has the potential to offer fine-grained control over the geometric structure captured in the diffusion process. Quantitative and qualitative improvements are demonstrated over baselines across several datasets.
summary: The paper modifies the original DDPM formulation by Ho et al., altering the noise schedule to concentrate in localized regions according to the gradient magnitude of the image $\frac{|| \nabla_{\mathbf{x}_0}||}{\lambda(t)}$ and edge sensitivity $\lambda(t)$, offering control over different frequencies modelled during training. This approach leads to up to improvement in FID over the DDPM baseline, and the results qualitatively demonstrate finer details and better structure can be modeled.
weaknesses: Substantial advances in the diffusion community have not been discussed or incorporated adequately within this paper. Therefore, it is not clear to the community whether this approach will offer significant benefit when scaling to larger problems or stronger baselines. In particular:

* The reported FIDs and qualitative results do not seem competitive with current state-of-the-art diffusion models. FIDs of around **26.15** for 128$\times$128 seem too high. Zhang et al., 2022, "Dimensionality-Varying Diffusion Process", attain CelebA 128^2 of around **~6**, $\infty-$Diff (Bond-Taylor et al., ICLR 2024) receives FID_CLIP on CelebAHQ of ~3, and recent diffusion papers attain ~2 on even more difficult datasets such as ImageNet (see the review in Hoogeboom et al., "Simpler Diffusion" Table 3, covering low FID 2023/2024 papers).
* Recent advances in Flow Matching (Lipman et al., ICLR 2023) offer a simpler theoretical framework, offering a direct (near optimal) path through the transition space—demonstrated in real-world settings such as StableDiffusion3. This could simplify the edge-preserving schedule, and make it easier to control than Ho el al., schedule derivation, while giving improved samples. Likelihood estimates can also be attained through FM. I would like to see discussion or comparison.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. “While the visual improvement over DDPM is subtle, our model generally demonstrated greater robustness to artifacts.” May I ask where can I see the experimentation results support this statement?
2. “In practice, we didn’t notice any increase in training/inference time.” May I ask what are the approximate numbers for the training/inference time?
soundness: 2
strengths: The authors proposed an interesting idea about how to keep the content clear and stable during the diffusion process, assuming the loss of detail content is caused by the isotropic noise scheme (like DDPM), by adopting existing research ideas from image processing area.
summary: The authors proposed a variant of DDPM, named content-aware diffusion models, putting more focus on the edges. They made some modifications on the noise scheme to make it edge-preserving noise, and edit the formula for forward and backward calculation as well.
weaknesses: 1. Experimentations might not be enough, especially in the quantitative comparisons with other existing works. Especially, if you want to state your proposed methods can be more content-aware compared to “isotropic diffusion models”, but not ONLY “DDPM”, then you have to compare it with other recent “isotropic diffusion models”. 
2. In addition, although with good explanation and math works, I’m feeling like this adaptation is more like adding an “weight matrix” to a specific focus, via an existing formula learned from image processing domain, during the diffusion process, and thus might not be impactful/novel enough. I would like to also hear other reviewers’ comments.

Meta-review: This paper introduces an edge-preserving diffusion model for image generation. It proposes an edge-aware noise scheduler that varies between edge-preserving and isotropic Gaussian noise, aiming to better capture structural information in images. The authors claim their model converges faster, better learns low-to-mid frequencies crucial for shape representation, and outperforms state-of-the-art baselines in unconditional image generation and tasks guided by a shape-based prior.    

The paper introduces a novel edge-preserving approach with a theoretical foundation that generalizes DDPM. It has potential for image generation and editing applications. The authors present both qualitative and quantitative improvements, including a reduction in FID scores. On the negative side, some reviewers mention that the paper's novelty is limited, as it primarily applies existing theory. Additionally, its performance is weak compared to current state-of-the-art methods, and it lacks comparisons with relevant work. The evaluation metrics used are limited, and there is a lack of discussion on generalization to other tasks.

The paper presents an interesting idea, but it is not ready for publication in its current form. The reviewers raise valid concerns regarding the novelty, performance, and evaluation of the proposed method. While the authors have partially addressed some concerns during the rebuttal phase, major issues remain. Specifically, the limited novelty, weak performance, and lack of comprehensive evaluation prevent this paper from meeting the ICLR acceptance threshold.

Decision: Reject