Edge-preserving noise for diffusion models

Authors: Jente Vandersanden, Sascha Holl, Xingchang Huang, Gurprit Singh
Primary area: generative models
Keywords: diffusion, generative modelling, denoising

Abstract:
Classical generative diffusion models learn an isotropic Gaussian denoising process,
treating all spatial regions uniformly, thus neglecting potentially valuable structural
information in the data. Inspired by the long-established work on anisotropic
diffusion in image processing, we present a novel edge-preserving diffusion model
that is a generalization of denoising diffusion probablistic models (DDPM). In
particular, we introduce an edge-aware noise scheduler that varies between edgepreserving
and isotropic Gaussian noise. We show that our model’s generative
process converges faster to results that more closely match the target distribution.
We demonstrate its capability to better learn the low-to-mid frequencies within the
dataset, which plays a crucial role in representing shapes and structural information.
Our edge-preserving diffusion process consistently outperforms state-of-the-art
baselines in unconditional image generation. It is also more robust for generative
tasks guided by a shape-based prior, such as stroke-to-image generation. We
present qualitative and quantitative results showing consistent improvements (FID
score) of up to 30% for both tasks.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730438446208

thread_type:
official_review

<thread_object>
creator: reviewer_q9qp

<content>
summary:
The authors proposed a variant of DDPM, named content-aware diffusion models, putting more focus on the edges. They made some modifications on the noise scheme to make it edge-preserving noise, and edit the formula for forward and backward calculation as well.

strengths:
The authors proposed an interesting idea about how to keep the content clear and stable during the diffusion process, assuming the loss of detail content is caused by the isotropic noise scheme (like DDPM), by adopting existing research ideas from image processing area.

weaknesses:
1. Experimentations might not be enough, especially in the quantitative comparisons with other existing works. Especially, if you want to state your proposed methods can be more content-aware compared to “isotropic diffusion models”, but not ONLY “DDPM”, then you have to compare it with other recent “isotropic diffusion models”. 
2. In addition, although with good explanation and math works, I’m feeling like this adaptation is more like adding an “weight matrix” to a specific focus, via an existing formula learned from image processing domain, during the diffusion process, and thus might not be impactful/novel enough. I would like to also hear other reviewers’ comments.

questions:
1. “While the visual improvement over DDPM is subtle, our model generally demonstrated greater robustness to artifacts.” May I ask where can I see the experimentation results support this statement?
2. “In practice, we didn’t notice any increase in training/inference time.” May I ask what are the approximate numbers for the training/inference time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730482399338

thread_type:
official_review

<thread_object>
creator: reviewer_wk9j

<content>
summary:
The paper modifies the original DDPM formulation by Ho et al., altering the noise schedule to concentrate in localized regions according to the gradient magnitude of the image $\frac{|| \nabla_{\mathbf{x}_0}||}{\lambda(t)}$ and edge sensitivity $\lambda(t)$, offering control over different frequencies modelled during training. This approach leads to up to improvement in FID over the DDPM baseline, and the results qualitatively demonstrate finer details and better structure can be modeled.

strengths:
The idea is intuitively simple and straightforward to incorporate into generative imaging applications. It has the potential to offer fine-grained control over the geometric structure captured in the diffusion process. Quantitative and qualitative improvements are demonstrated over baselines across several datasets.

weaknesses:
Substantial advances in the diffusion community have not been discussed or incorporated adequately within this paper. Therefore, it is not clear to the community whether this approach will offer significant benefit when scaling to larger problems or stronger baselines. In particular:

* The reported FIDs and qualitative results do not seem competitive with current state-of-the-art diffusion models. FIDs of around **26.15** for 128$\times$128 seem too high. Zhang et al., 2022, "Dimensionality-Varying Diffusion Process", attain CelebA 128^2 of around **~6**, $\infty-$Diff (Bond-Taylor et al., ICLR 2024) receives FID_CLIP on CelebAHQ of ~3, and recent diffusion papers attain ~2 on even more difficult datasets such as ImageNet (see the review in Hoogeboom et al., "Simpler Diffusion" Table 3, covering low FID 2023/2024 papers).
* Recent advances in Flow Matching (Lipman et al., ICLR 2023) offer a simpler theoretical framework, offering a direct (near optimal) path through the transition space—demonstrated in real-world settings such as StableDiffusion3. This could simplify the edge-preserving schedule, and make it easier to control than Ho el al., schedule derivation, while giving improved samples. Likelihood estimates can also be attained through FM. I would like to see discussion or comparison.

questions:
* It is not clear whether NLLs can be approximated easily with this approach, potentially limiting the applications. Please clarify or demonstrate this.
* The time-varying edge sensitivity $\lambda(t)$ seems arbitrary in section 5, and is currently dependent on Ho et al.,'s schedule, tying this down to similar variants. 
   * Can the main theory be better abstracted and demonstrated in different diffusion frameworks?
   * Is the approach suited for Flow Matching?
* Implementing this within a modern diffusion framework/architecture that efficiently supports high-resolution datasets would be beneficial to quantitatively evaluate improvements.
   * How does this scale with FFHQ $1024^2$ or ImageNet-$128$?
* Please clarify how $|| \nabla_{\mathbf{x}_0}||$ is implemented. Is this approximated with a Sobel filter?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518842383

thread_type:
official_review

<thread_object>
creator: reviewer_muvy

<content>
summary:
This paper introduces a novel edge-preserving diffusion model that generalizes denoising diffusion probabilistic models (DDPMs). By incorporating an edge-aware noise scheduler, the model retains structural information in images, converges more rapidly to the target distribution, and better captures low-to-mid frequency components within datasets, which are crucial for representing shapes and structural details.

strengths:
This paper proposes an interesting approach to improve the generation quality of diffusion models.
The experiments are extensive, and the visualizations are clear, helping readers understand the improved generation performance.

weaknesses:
1. Does edge-aware noise affect generation diversity?
2. In the "Related Work" section, you initially describe IHDM as an isotropic Gaussian blurring model, yet later refer to it and BNDM as incorporating a form of non-isotropic noise. This creates a seeming contradiction that would benefit from further clarification to ensure the logical coherence of your manuscript.
3. While the paper makes a commendable effort to integrate the anisotropic diffusion concept from Perona and Malik (1990) into the diffusion model framework, the innovation appears to be primarily an application of existing theory rather than a novel theoretical contribution. The incorporation of edge-preserving noise as a new term in the diffusion process, while potentially enhancing the model's performance in certain aspects, does not fundamentally alter the underlying principles of generative diffusion models.
4. The discussion surrounding Eq. (2) and Eq. (3) in the paper is somewhat perplexing due to its circuitous approach. The authors explore the relationship between diffusion models and heat conduction, as well as the limitations of isotropic noise, before returning to the contributions of Perona and Malik (1990) in Eq. (10). A more effective strategy would be to engage with Perona and Malik's work on edge-preserving diffusion earlier in the discussion. This would create a natural transition to Eq. (12), which introduces the paper's core innovation—a novel noise term.
5. In the "Background" section, when discussing the "Forward and backward process posteriors," you utilize the formulation presented in Eq. (1), which is consistent with the isotropic Gaussian process. However, you also reference Eq. (2), which is derived from the heat dissipation process and represents a different formulation. My concern is that the use of both Eq. (1) and Eq. (2) in this context may create confusion, especially since your novel edge-preserving noise scheme seems to be more aligned with the principles of Eq. (1), as inferred from the derivations in Section 4.2. It appears that your improved noise term is inspired by the anisotropic diffusion work of Perona and Malik, which is more closely related to Eq. (2). Yet, the parameter calculations and posterior calculations in your model follow a form similar to Eq. (1). If this is the case, then the inclusion of Eq. (2) in the introduction might be seen as unnecessary, and it could potentially detract from the clarity of your paper. I would recommend either providing a more explicit connection between your noise model and Eq. (2), or if Eq. (2) is not central to your approach, consider streamlining the presentation to focus on Eq. (1) and its relevance to your edge-preserving noise scheme. This would help readers to better understand the relationship between your work and the existing literature and clarify why both equations are presented if they are not both integral to your model.

questions:
Please refer to the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643750786

thread_type:
official_review

<thread_object>
creator: reviewer_vzw3

<content>
summary:
The paper introduces a novel approach to generative diffusion models by incorporating edge-preserving noise, which is a generalization of denoising diffusion probabilistic models (DDPM). The authors propose an edge-aware noise scheduler that alternates between edge-preserving and isotropic Gaussian noise, aiming to better capture the structural information in data. The model demonstrates faster convergence to the target distribution and improved learning of low-to-mid frequencies, which are crucial for shape and structural representation.

strengths:
1. The paper provides a theoretical foundation for the proposed model, explaining how it generalizes DDPM and improves frequency learning.

2. The model's ability to better preserve structural information could have practical benefits in various image generation and editing applications.

3. The paper presents a creative method for improving diffusion models by introducing edge-preserving noise, addressing a challenge in generative modeling.

4. The method shows substantial improvements in both qualitative and quantitative metrics, particularly in reducing FID scores and enhancing image generation quality.

weaknesses:
1. The paper does not discuss the computational cost of the proposed method compared to existing approaches, which is an important consideration for practical applications. Suggesting comparisons with existing methods on these metrics could provide valuable context. For instance, a table comparing memory usage and speed (inference and training) across baseline models could highlight this model's practical viability more clearly.

2. While the paper demonstrates improvements on a specific dataset, it is unclear how well the model generalizes to other types of models or tasks. Testing on tasks that emphasize diverse structural features, such as denoising or inpainting, could provide insight into how well the model maintains performance across broader applications.

3. The current evaluation uses limited metrics to assess the effectiveness of the proposed method. Including newer metrics like CLIPScore, etc. could provide a more comprehensive view of performance. Specifically, CLIPScore could offer insights into the model's ability to generate semantically consistent images. Given the model's edge-preserving focus, metrics that capture detail preservation and structure alignment might also be beneficial.

4. While the paper compares the proposed method with existing baselines, it could benefit from a more detailed comparative analysis with other state-of-the-art methods, especially those that also aim to preserve structural information in images.

5. The performance of the proposed model may be highly sensitive to the choice of hyperparameters, such as the transition function and edge sensitivity. The paper could provide **more insight** into how these parameters affect the model's performance and **how to select** them effectively.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1732313828133

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response to all reviewers (part 1)

comment:
Dear reviewers, thank you for your valuable feedback and suggestions. This global comment addresses shared concerns and summarizes additional results/comparisons. We are preparing an updated paper with color-coded changes for clarity, to be uploaded later in the rebuttal phase.

## 1. Can our edge-preserving model be made more abstract, so it can be used in the context of other diffusion frameworks? Is it general enough?
Within the diffusion community there are 2 major "paradigms": score-based models and denoising probabilistic models. We show how our diffusion process is rather general and fits in both paradigms, theoretically making it compatible with existing applications located within these paradigms. 

Everything starts from the linear equation Eq. (1) in our submission. To make our process edge-preserving, we choose a $\sigma_t$ that suppresses noise, according to the coefficient $\frac{u}{(1-\tau(t)) \sqrt{1 + \frac{||\mathbf{x}_0||}{\lambda(t)}} + \tau(t)}$. For $\gamma_t$ and the numerator $u$, we remained consistent with DDPM's signal and noise coefficients, because the goal of our paper is to measure the mere impact of edge-preserving (non-isotropic) noise on the performance of a generative diffusion model. But nothing prevents us from choosing other values for $\gamma_t$ and $u$.

Given this linear equation, we can choose a paradigm: 

**In our paper we follow the denoising probabilistic paradigm:**
Because our diffusion process is linear, we end up with a well-studied Gaussian process with known posteriors, introduced in Section 3 of the paper. By updating the backward posteriors according to our edge-preserving process (Eq. (14) - (16) in our submission) we can generate samples by simulating the backward Gaussian process.

**But, our process also fits in the score-based modelling paradigm:**
We can rewrite our Eq. (13) as the SDE: $d\mathbf{y}_t = \hat{b}(t)dt + \hat{\varsigma} d \mathbf{w}_t$ , where $\hat{b}(t) = \sqrt{\bar{\alpha}_t}$ and $\hat{\varsigma}(t) = \frac{\sqrt{1 - \bar{\alpha}_t}}{(1 - \tau(t))\sqrt{1 + \frac{||\nabla \mathbf{x}_0||}{\lambda(t)}} + \tau(t)}$. We will provide a more detailed derivation and discussion in Appendix B of the revised version of the paper

Briefly said, our model fits in both general diffusion paradigms, and still leaves possibility for many different variants of edge-preserving  processes (e.g. with different $\gamma_t$'s, different numerators of $\sigma_t$). We will be more explicit about this in the revised version which we will submit later in this rebuttal phase. Hopefully this addresses the concern of reviewer mUvY and reviewer WK9j about whether our formulation is general enough and/or tied to existing models. 

## 2. How is the method different from DDPM? There seem to be similarities, which might introduce dependencies on DDPM.
As discussed in the paragraph above, our diffusion process fits into (at least) two general widely-adopted paradigms of generative diffusion models. We clarify that our method is not tied to DDPM. While we follow the same paradigm, keep consistent with their signal coefficient $\gamma_t$, and include their $\sigma_t$ in our $\mathbf{\sigma}_t$, this was to solely study the effect of non-isotropic, content-aware noise on a well-studied isotropic diffusion model. That is, we wanted to keep as many parameters consistent as possible, and only measure the impact of edge-preserving noise. Despite that, an infinite amount of choices of the parameters $\gamma_t$ and $\sigma_t$ from Eq. (1) are possible, and nothing prevents us from making completely different choices. Our edge-preserving diffusion process is a process living in the same paradigm, but independent from DDPM.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732313891877

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response to all reviewers (part 2)

comment:
## 3. Additional quantitative comparisons with Simple Diffusion (Hoogeboom et al. 2023)
Below we provide comparisons on shape-guided and unconditional image generation with an additional state of the art isotropic baseline, Simple Diffusion (Hoogeboom et al. 2023) [6]. Unfortunately, there is no official implementation made available by the authors, therefore we resorted to this [open-source implementation of Simple Diffusion](https://github.com/faverogian/simpleDiffusion). Note that this implementation does not consider the architectural changes proposed in the paper. Consistent with the results in our submission, our edge-preserving model consistently outperforms Simple Diffusion on the shape-guided generative task based on SDEdit (Meng et al. 2021) [1]. 

| FID/CLIP score - Shape-guided image generation (SDEdit (Meng et al. 2021))| Ours                  | Simple Diffusion  | 
|---------------------------------------------------------------------------|-----------------------|-------------------|
| AFHQ-Cat($128^2$)                                                         | **23.50** / **88.97** | 34.26 / 88.23     |                          
| CelebA($128^2$)                                                           | **39.08** / **61.15** | 46.43  / 60.72    |                            
| LSUN-Church($128^2$)                                                      | 56.14 / **64.32**     | **45.93** / 62.13 | 

To isolate the impact of edge-preserving noise, it’s fairer to compare the original Simple Diffusion with an edge-preserving version. While we’re still working on this integration due to time constraints, we provide results based on our original setup presented in the submission. Although we get outperformed in unconditional image generation, we are optimistic that incorporating edge-preserving noise into Simple Diffusion will enhance its performance, as we saw with DDPM. We contend that these are reasonable expectations, as both our method and theirs align with the probabilistic framework of known Gaussian processes outlined in the previous paragraph.

| FID - Unconditional image generation             | Ours     | Simple Diffusion |
|--------------------------------------------------|-----------|------------------|
| AFHQ-Cat($128^2$)                                | **13.06** |     15.66        |
| CelebA($128^2$)                                  | 26.17     |     **19.28**    |
| LSUN-Church($128^2$)                             | 23.17     |     **17.87**    |

**Why are our reported FIDs in the double-digits, which seems higher than what state-of-the-art baselines report?**
We trained all baselines and our model for the same number of epochs within a 48-hour, 2x NVIDIA Tesla A40 GPU budget, reflecting real-world training constraints. Using a simple U-Net, our goal was to evaluate the impact of edge-preserving noise, not advanced architectures. With more sophisticated architectures like Simple Diffusion's, we expect FID scores to improve. To motivate the impact of training budget and model architecture, we ran an [open-source implementation of Simple Diffusion](https://github.com/faverogian/simpleDiffusion) with our simple U-Net and found similar double-digit FID scores.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732313990709

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response to all reviewers (part 3)

comment:
## 4. Timings for inference and training
Our measurements are based on data resolution (128x128) and a batch size of 64. Note that BNDM and Flow Matching make use of less inference steps (T=250 vs. T=500 for Ours, DDPM and Simple Diffusion), and therefore are expected to be faster for inference. Given that the official implementations of Simple Diffusion (Hoogeboom et al. 2023) and Flow Matching (Lipman et al. 2023) are unavailable, we used [simpleDiffusion](https://github.com/faverogian/simpleDiffusion) and [torchcfm](https://github.com/atong01/conditional-flow-matching) respectively. Our setup consisted of 2 NVIDIA Quadro RTX 8000 GPUS. We see that timings and memory usage of Ours is very similar to DDPM and Simple Diffusion, suggesting that the Sobel filter we apply to approximate $||\nabla \mathbf{x}_0||$ brings minimal overhead. We will add these measurements to the revised version of the paper which we will upload later during the rebuttal phase.


Training:
|                              | Ours | DDPM | BNDM | Flow Matching (Lipman et al. 2022) | Simple Diffusion (Hoogeboom et al. 2023) |
|------------------------------|------|------|------|------------------------------------|------------------------------------------|
| **Time/training iteration in s** | 1.12 | 1.11 | 0.74 | 2.74                               | 1.48                                     |


 Inference:
|                                 | Ours  | DDPM  | BNDM | Flow Matching (Lipman et al. 2022) | Simple Diffusion (Hoogeboom et al. 2023) |
|---------------------------------|-------|-------|------|------------------------------------|------------------------------------------|
| **Inference time in s for 1 batch** | 301.5 | 277.5 | 77.2 | 84.78                              | 290.7                                    |

|                          | Ours | DDPM | BNDM | Flow Matching (Lipman et al. 2022) | Simple Diffusion (Hoogeboom et al. 2023) |
|--------------------------|------|------|------|------------------------------------|------------------------------------------|
| **Total memory usage in GB** | 9.16 | 9.16 | 10.3 | 22.18                              | 10.42                                    | 

## Additional comparisons on shape-guided generative task with CLIPScore metric
The table below provides an additional comparison for our shape-guided generative task (cf. SDEdit, Meng et al., 2021) evaluated using the CLIPScore metric. Our method consistently outperforms the baselines on this metric, indicating that the generated images are more semantically aligned with the ground-truths (the original images used to generate the stroke paintings). In the submission, we showed several examples (e.g. Figure 5) where our model solves visual artifacts that are apparent with other baselines, which can improve the semantical meaning of the generated image.

| CLIP score for shape-guided generative task (SDEdit) | Ours      | DDPM  | Simple Diffusion |
|------------------------------------------------------|-----------|-------|------------------|
| AFHQ-Cat($128^2$)                                    | **88.97** | 88.78 | 88.23            |
| CelebA($128^2$)                                      | **61.15** | 61.02 | 60.72            |
| LSUN-Church($128^2$)                                 | **64.32** | 62.57 | 62.13            |

References:
[1]: [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, Meng et al. 2021](https://arxiv.org/abs/2108.01073)
[2]: [RePaint: Inpainting using Denoising Diffusion Probabilistic Models, Lugmayr et al. 2022](https://arxiv.org/abs/2201.09865)
[3]:[Generative Modelling With Inverse Heat Dissipation, Rissanen et al. 2023](https://arxiv.org/pdf/2206.13397)
[4]:[Generative Modeling by Estimating Gradients of the Data Distribution
, Song et al. 2019](https://arxiv.org/abs/1907.05600)
[5]:[Score-Based Generative Modeling through Stochastic Differential Equations](https://arxiv.org/abs/2011.13456)
[6]: [Simple Diffusion: End-to-end diffusion for high resolution images, Hoogeboom et al. 2023](https://arxiv.org/abs/2301.11093)
[7]: [Flow Matching for Generative Modeling
, Lipman et al. 2023](https://arxiv.org/abs/2210.02747)
[8]: [Denoising Diffusion Probabilistic Models: Ho et al. 2020](https://arxiv.org/abs/2006.11239)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732706962955

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision submission

comment:
Dear reviewers,

Thank you all for your constructive feedback in helping us improve the paper. While we are still refining the paper, we wanted to share the current version to illustrate how we plan to incorporate the reviewers' suggestions. We would greatly appreciate any additional feedback to further enhance the revised version.

We have uploaded the revised version, incorporating the following changes (highlighted in blue):

- We have updated the Background section (following suggestions from reviewer mUvY) by introducing discussions on how our edge-preserving model can adapt to other generative paradigms like SDE or score-based generative models (details in Appendix A). This would improve the readability and make a smoother transition towards our formulating based no probabilistic models. 
- Added theoretical formulation regarding the relation to Flow Matching (Lipman et al. 2022) in Appendix B, as well as a comparison to Flow Matching in Table 2 (following suggestions from the reviewer WK9j).
- Following WK9j concerns, we explain how negative log-likelihood can be approximated with our formulation in Appendix D.
- We have rewritten equation 11 and the corresponding text for Section 4.1 (following suggestions from the reviewer mUvY).
- Timings and memory consumptions are added in Table 6, showing our method brings minimal overhead compared to DDPM (following suggestions from the reviewer Q9qP).
- We have added additional comparison with Simple Diffusion in Table 4, 5 (following suggestions from the reviewers Q9qP, WK9j).
- Table 7 and 8 are added to perform realism and diversity analysis (following suggestions from the reviewer mUvY).
- We also add new metric comparison in Table 10 with the CLIP score for shape guided image generation (SDEdit) (following suggestions from the reviewers WK9j, Q9qP, vZw3).
- In Table 9/Figure 14, we incorporate our Edge-preserving noise into RePaint (Lugmayr et al., 2022) for image inpainting with visual comparisons (following suggestions from the reviewer vZw3).

Thank you,
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces an edge-preserving diffusion model for image generation. It proposes an edge-aware noise scheduler that varies between edge-preserving and isotropic Gaussian noise, aiming to better capture structural information in images. The authors claim their model converges faster, better learns low-to-mid frequencies crucial for shape representation, and outperforms state-of-the-art baselines in unconditional image generation and tasks guided by a shape-based prior.    

The paper introduces a novel edge-preserving approach with a theoretical foundation that generalizes DDPM. It has potential for image generation and editing applications. The authors present both qualitative and quantitative improvements, including a reduction in FID scores. On the negative side, some reviewers mention that the paper's novelty is limited, as it primarily applies existing theory. Additionally, its performance is weak compared to current state-of-the-art methods, and it lacks comparisons with relevant work. The evaluation metrics used are limited, and there is a lack of discussion on generalization to other tasks.

The paper presents an interesting idea, but it is not ready for publication in its current form. The reviewers raise valid concerns regarding the novelty, performance, and evaluation of the proposed method. While the authors have partially addressed some concerns during the rebuttal phase, major issues remain. Specifically, the limited novelty, weak performance, and lack of comprehensive evaluation prevent this paper from meeting the ICLR acceptance threshold.