Title: Constructing Non-isotropic Gaussian Diffusion Model Using Isotropic Gaussian Diffusion Model for Image Editing
Venue: NeurIPS 2023
Accepted: yes
Keywords: score-based diffusion model

non-isotropic Gaussian diffusion model

image editing
Abstract: Score-based diffusion models (SBDMs) have achieved state-of-the-art results in image generation. In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing, which requires editing the source image while preserving the image regions irrelevant to the editing task. We construct NGDM by adding independent Gaussian noises with different variances to different image pixels. Instead of specifically training the NGDM, we rectify the NGDM into an isotropic Gaussian diffusion model with different pixels having different total forward diffusion time. We propose to reverse the diffusion by designing a sampling method that starts at different time for different pixels for denoising to generate images using the pre-trained isotropic Gaussian diffusion model. Experimental results show that NGDM achieves state-of-the-art performance for image editing tasks, considering the trade-off between the fidelity to the source image and alignment with the desired editing target.

Constructing Non-isotropic Gaussian Diffusion Model Using Isotropic Gaussian Diffusion Model for Image Editing

Abstract

Score-based diffusion models (SBDMs) have achieved state-of-the-art results in image generation. In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing, which requires editing the source image while preserving the image regions irrelevant to the editing task. We construct NGDM by adding independent Gaussian noises with different variances to different image pixels. Instead of specifically training the NGDM, we rectify the NGDM into an isotropic Gaussian diffusion model with different pixels having different total forward diffusion time. We propose to reverse the diffusion by designing a sampling method that starts at different time for different pixels for denoising to generate images using the pre-trained isotropic Gaussian diffusion model. Experimental results show that NGDM achieves state-of-the-art performance for image editing tasks, considering the trade-off between the fidelity to the source image and alignment with the desired editing target.

1 Introduction

Score-based diffusion models (SBDMs) [1-6] demonstrate state-of-the-art performance on image synthesis quality and sample diversity. SBDMs are widely applied to applications such as text-toimage synthesis [7-9], image editing [10-15], deblurring [16, 17], etc. SBDMs consist of a forward diffusion stage that adds random noise to data and a reverse stage that generates desired data from noise. The introduced noise in the forward process is commonly isotropic Gaussian noise [1, 6], i.e., independently and identically distributed noise in a normal distribution.

Non-isotropic diffusion model, by adding non-isotropic noises in the forward diffusion process has been investigated in [18-22]. The blurring diffusion model in [18] adds blur and noise to samples which is a Gaussian diffusion process with non-isotropic noise in the frequency space. [19] employs auxiliary velocity variables to augment the data variables as Hamiltonian dynamics, and performs the diffusion process in this expanded joint space by adding different noises to the auxiliary and the data variables. [20] formulates the diffusion model using non-isotropic noise with a positive semi-definite covariance matrix, and carries out a comparative analysis of the non-isotropic and isotropic diffusion models. These works have shown better performance for data generation.

This paper focuses on image editing tasks that commonly require editing specific object/thing of an image while preserving the remaining parts of the image. For image editing, [11] produces a mask that allows the preservation of context while editing the remaining part. [23-25] use the learned text embedding for the object that needs to be preserved in the image to ensure the object is unchanged during editing. It is empirically known that the diffusion model can generate more diverse novel

content if adding noise with larger variance to the image while preserving the image content if adding smaller variance noise [13]. Motivated by this, we employ a non-isotropic diffusion model to add noises with different variances to different image pixels, considering the degree to which the corresponding pixels should be edited/preserved.

Along with this idea, we present a Non-isotropic Gaussian Diffusion Model (NGDM), utilizing an off-the-shelf isotropic diffusion model (e.g., DDPM [1]) for achieving the data sampling in image editing. Specifically, given a source image, the proposed NGDM is with a diffusion process that adds independent Gaussian noises with different variances to different pixels, therefore the added noise is independent and non-isotropic over image pixels. We then employ an off-the-shelf isotropic diffusion model to execute the reverse denoising process for image editing. To achieve this goal, we rectify the NGDM into the isotropic Gaussian diffusion model where each pixel is added with the same amount of noise at each step but different pixels have varying total number of noise accumulation time steps. We subsequently devise a specific sampling method for NGDM that can generate images by using the pre-trained isotopic Gaussian diffusion model.

We demonstrate the effectiveness of NGDM in image editing tasks on five datasets including real and synthetic datasets. In the experiments for cats to dogs editing task on AFHQ dataset, our method achieves the best performance in the metric of FID and SSIM compared with the SoTA SBDMs-based methods. For text-guided image editing, our method achieves better trade-off between CLIPScore and LPIPS value. Furthermore, NGDM allows for flexible trade-off with varying hyper-parameters.

2 Background: Score-based Diffusion Models

SBDMs [1, 2, 4-6] are a family of generative models that learn the data distribution based on the Gaussian process. Two representative models are Denoising Diffusion Probabilistic Model (DDPM) [1] and Score Matching with Langevin Dynamics (SMLD) [5]. We discuss the details based on DDPM for the remainder of the main text for brevity.

Given the input data x (0) ∈ R D , which represents a sample from the data distribution p data , a forward process produces the noisy x ( t ) indexed by a time variable t ∈ [0 , 1] via

where z ( t ) ∈ N ( 0 , I ) for any t and ¯ α ( t ) = e -∫ t 0 β ( s )d s controlling the noise schedule. β ( s ) = ¯ β min + s ( ¯ β max -¯ β min ) with ¯ β min = 0 . 1 and ¯ β max = 20 [1, 6]. This type of diffusion model is dubbed Isotropic Gaussian Diffusion Model (IGDM) , since the added Gaussian noise z ( t ) is from the independently and identically distributed normal distribution.

DDPMis in the framework of Stochastic Differential Equations (SDEs) [5] with variance preservation

where w is the standard Wiener process. The reverse process denoises the noisy sample x ( T ) starting from T using a reverse SDE

where ¯ w is a standard Wiener process when time flows backward from T to 0 . The score function ∇ x log p t ( x ) is approximated by training a time-dependent model s θ ( x ( t ) , t ) via score matching [6, 26]. For inference, the time of the differential equation is discretized as t ∈ { 0 , ∆ t, 2∆ t, · · · , T } with ∆ t representing the sampling time interval. We can choose to utilize the reverse process of DDPM or the reverse process of DDIM for sampling. With β t = β ( t )∆ t , the iteration rule of DDPM [1] is

where z ( t +∆ t ) ∈ N ( 0 , I ) . With ¯ α t = ∏ t s =0 (1 -β s ) , the iteration rule of DDIM [27] is

!

0

Forward

Reverse

0

IGDM

! #

!$

!%

! #

!$

!%

Source

image

Translated

image

NGDM

Translated

image

y(0)

NGDM

3 Method

In this section, we present a framework for utilizing the pre-trained Isotropic Gaussian Diffusion Model (IGDM) to achieve the sampling process of Non-isotropic Gaussian Diffusion Model (NGDM). In the following, first, we define the NGDM with added independent non-isotropic Gaussian noise. Then, we implement NGDM by IGDM through rectifying the spatially different time of noising and denoising procedures in NGDM and present our proposed data sampling algorithm for the proposed NGDMusing the pre-trained IGDM.

3.1 Non-isotropic Gaussian Diffusion Model

In this work, we focus on the Non-isotropic Gaussian Diffusion Model (NGDM) by adding the nonisotropic Gaussian noise in the input data y (0) ∈ R D and y (0) ∼ p data , and the noises associated with different pixels are independent. The forward SDE of NGDM is

where I ∈ R D is the source data, Λ ( I ) : R D → R D × D is the weighting coefficient matrix, defined as diagonal matrix Λ ( I ) = diag ( λ 1 , · · · , λ D ) with 0 ≤ λ k ≤ 1 scaling the Gaussian noise level added to the k -th pixel. Note that the transition kernel p 0 t ( y ( t ) | y (0)) = N ( y ( t ) | y (0) e -1 2 ∫ t 0 β ( s ) Λ ( I )d s , I -e -∫ t 0 β ( s ) Λ ( I )d s ) is an independent Gaussian distribution.

3.2 Rectify the Non-isotropic Gaussian Diffusion Model

With the added independent noise, we next discuss the forward SDE for NGDM in scalar form for each pixel k . Given the initial y k (0) denoting the value of pixel k in y (0) , the forward SDE of the k -th pixel can be presented by

where w is a one-dimensional Wiener process.

We present Theorem 1 to illustrate the connection between the NGDM defined in Section 3.1 and the IGDM defined in Section 2 at the pixel level. Beforehand, we introduce the following lemma.

Lemma 1. Let β ( s ) = ¯ β min + s ( ¯ β max -¯ β min ) with ¯ β max > ¯ β min > 0 . Then, for each λ k ∈ [0 , 1] and t ∈ [0 , 1] , there exists a unique time τ ∈ [0 , 1] (denoted by τ = ξ k ( t ) ) such that ∫ t 0 β ( s ) λ k d s = ∫ τ 0 β ( s )d s and β ( t ) λ k d t = β ( τ )d τ , with the following form

The proof is in Appendix A. Based on the above Lemma, we can rectify the NGDM, which adds noise at each pixel with varying variance over the same time span, into an IGDM that adds noise at each pixel with the same noise variance but with different total diffusion time for different pixels. We introduce the following theorem to derive the differential equation as an IGDM.

Theorem 1. For a pixel indexed by k , λ k ∈ [0 , 1] , and let τ = ξ k ( t ) with ξ k ( t ) represented in Eq. (8) . With the same initial value y k (0) , we have that the transition kernel at time t induced by Eq. (7) is equal to the transition kernel at time τ induced by the following differential equation

The total time of noising for Eq. (9) is T k with T k = ξ k ( T ) .

We provide the proof in Appendix A. Inspired by this, we rectify the reverse process in NGDM with different speeds of denoising across pixels to be the reverse process with consistent speed but different total time of denoising. We suggest rectifying the differential equation for the reverse process of pixel k within the NGDM framework into the following form

where ¯ w is a one-dimensional Wiener process when time flows backward from T k to 0 . Theorem 1 establishes the conclusion that the NGDM in Eq. (7) can be rectified to the IGDM in Eq. (9) but with different total diffusion time T k for different pixel indexed by k , determined based on Eq. (8). This inspires us to utilize the pre-trained IGDM to achieve the data sampling of NGDM for image editing. Subsequently, we present a method that adjusts the total time of noising and denoising for each pixel k to T k , enabling us to use the pre-trained IGDM for data sampling. The corresponding sampling method is presented in Algorithm 1.

For image editing tasks, we use the source image I as y (0) and generate noisy data y ( T ) through the forward process. We generate the edited image ˆ y (0) by denoising from y ( T ) . Utilizing the forward noising process of IGDM, we add independent noise to each pixel k to obtain the noisy observation x k ( t ) of discrete time t ∈ { 0 , ∆ t, · · · , T } with ∆ t representing the sampling time interval

where z ( t ) ∈ N ( 0 , I ) . Next, with H ( y ( t +∆ t ) , t +∆ t ) denotes the sampling procedure of DDPM and DDIM given in Eq. (4) and Eq. (5) of Section 2, the data sampling iteration utilizing the IGDM model with initial value y k ( T ) = x k ( T ) is defined as

This implies that we use the noisy observation x k ( t ) to represent y k ( t ) at each step before T k with t ≥ T k , rather than the actual denoised result starting from time step T . Until time T k we begin the denoising from x k ( T k ) for k -th pixel. In such a way, different pixels have different starting time steps ( T k for k -th pixel) for image denoising in the data sampling process.

3.3 Sampling Method in NGDM

Based on the above method, we further specify our sampling algorithm by utilizing a pre-trained IGDM. We do not require the training of NGDM, but instead harness the power of a pre-trained IGDM for data sampling. We generate the edited image with the source image I as a condition. We first add noise to the source image I to T time steps as the starting point of denoising, and then use the method in Section 3.2 to rectify NGDM into IGDM to denoise the image using the pre-trained IGDM. We show the sampling algorithm of NGDM in the Algorithm 1.

Algorithm 1 Sampling Method of NGDM

Inputs: The source image I , the weighting matrix Λ ( I ) , the score function s θ , the time schedule { β ( t ) } T , the maximal time steps T

t =0 1: Compute ¯ T = [ ξ 1 ( T ) , · · · , ξ D ( T )] according to Eq. (8) 2: z ∼ N ( 0 , I ) 3: y = √ ¯ α T I + √ 1 -¯ α T z # The starting point of denoising 4: for t = T to 0 do 5: x = √ ¯ α t I + √ 1 -¯ α t z # Sample the noisy source image at time step t 6: M = I ( t ≥ ¯ T } ) 7: y ← M ⊙ x +( 1 -M ) ⊙ y # 1 is the D -dimensional all-ones vector 8: y ← H ( y , t ) 9: end for Output: Generated image y conditioned on the source image I

3.4 Design of Input-dependent Weighting Matrix

We construct the weighting matrix Λ ( I ) based on the method for designing mask using a textconditioned diffusion model [9] in DiffEdit [11]. Specifically, given the source image I , the text description R of the source image, and the target description Q that describes the desired target image after editing. Following DiffEdit [11], we add noise to the source image up to 0 . 5 T step and use the texts R and Q as the conditions respectively for denoising in the current time step to estimate the score values by using the score network s θ . We derive an attention map A ( I ) based on the absolute difference of the estimated scores. We use the above method to compute 10 estimated absolute noise differences by running 10 times with different random seeds, averaging and performing Gaussian smoothing on the averaged map. Finally, we normalize the values of the smoothed map to [0 , 1] as the final attention map A ( I ) . The pixel with larger value in the attention map should be added with the noise with larger variance, to sufficiently edit the content of the pixel. The pixel with smaller value in the attention map should be added with noise having smaller variance to preserve the content of pixel. We transform the attention map A ( I ) into the weighting matrix Λ ( I ) through a Sigmoid function by Λ ( I ) = 1 1+exp( -( a A ( I ) -b )) , where a and b are the hyper-parameters for the transformation. We discuss the effect of hyper-parameters a and b on the generation of images in Section 4.

4 Experiment

In this section, we apply the proposed NGDM to image editing tasks, presenting both qualitative and quantitative results.

4.1 Experimental Setup

Evaluation tasks. We perform experiments on five datasets. First, we experiment on AFHQ dataset [28] to edit cats into dogs. The source domain contains 500 images of cats. These images are resized to 256 × 256 resolution and subsequently edited into dogs. Second, we experiment on ImageNet dataset [29] to edit images from one class into another class based on text prompts, following the protocol of [30]. Third, we experiment on synthetic Imagen [31] dataset following DiffEdit [11]. We collect images generated by Imagen [31], along with their corresponding text prompts, and edit these images by altering portions of the text. Fourth, we experiment on COCO-S dataset and construct target prompts for editing from annotations provided by BISON [32]. We collect 1000 images and target prompts from the COCO [33] dataset to build the COCO-S dataset. We additionally consider DreamBooth dataset provided by [24], which contains 30 objects. Each object has 25 prompts and 4-6 images for editing. We edit each image using the provided 25 prompts, resulting in a total of 3950 edited images. More details are available in Appendix B.1.

Implementation details. We conduct experiments utilizing two types of pre-trained diffusion models. For the cats to dogs editing task on AFHQ dataset, we utilize the public pre-trained scorebased diffusion model with the official code provided in ILVR [10]. This model operates directly in the image space. We set the denoising step N to 1000. We implement the remaining task based on the text-to-image latent diffusion model, i.e., Stable Diffusion [9]. This model was pre-trained on 512

× 512 images of LAION dataset [34] with a latent dimension of 64 × 64. We use 50 steps in DDIM sampling with a fixed noise schedule. For hyper-parameters a and b , we set them to 10 . 0 and 5 . 0 respectively to obtain all qualitative comparison results presented in this paper. We conduct additional analysis to investigate the effect of different values of hyper-parameters in the experimental results. Unless otherwise stated, we use the default parameters of the diffusion model during inference.

Source

Ours

EGSDE [14]

SDEdit [13]

ILVR [10]

4.2 Results and Analysis

Results on AFHQ dataset. We report the widely-used Frechet Inception Distance (FID) [35] for quantifying realism and SSIM [36] for quantifying faithfulness. The quantitative comparisons and qualitative results are presented in Table 1 and Figure 2, respectively. The images generated by our method better preserves the contextural structure ( e . g ., pose) of cats while yielding realistic dog images. For example, Figure 2 shows that for images in columns 2-5 with complex backgrounds, our method can accurately keep the backgrounds unchanged while editing cats into dogs. Other methods either blur the backgrounds or fail to maintain the source image backgrounds correctly. This observation is further supported by the quantitative results in Table 1, where our method achieves the best results of FID and SSIM among the compared SBDM-based methods.

Source

Oystercatcher

↓

Flamingo

Broccoli

↓

Butternut

squash

Convertible

↓

Ambulance

Custard apple

↓

Lemon

Bolete

↓

Coral fungus

Kite

↓

Ours

DiffEdit

DiffuseIT

DDIB

SDEdit

Aura

Source

Ours

DiffEdit

SINE

SDEdit beach

↓

mountain fuzzy panda

↓

raccoon cat

↓

Shiba Inu dog mountain

↓

beach sunglasses

↓

hat

skateboarding

↓

riding a bike

(a)

(b)

(c)

(d)

Ours(10.0,5.0)

DiffuselT

InstrucPix2Pix

30.5

SDEdit

36.0-

30.0

35.8 -

SDEdit

CLIPScore

29.5

Ours (10.0,6.0)

DDIB

CLIPScore

35.6 -

Ours (10.0,6.0)

Ours(10.0,5.0)

InstructPix2Pix

35.4

29.0 -

DiffEdit

35.2

DiffEdit

28.5

35.0 -

DDS

28.0

SINE

EDICT

34.8

1

EDICT

DDS

27.5 -

30

35

40

45

50

55

60

65

20

25

30

35

40

45

50

LPIPS

LPIPS

Ours(10.0,5.0)

DDS

31.5-Ours (10.0,6.0)

lnstructPix2Pix

20.1 -

31.0

EDICT

20.0-

CLIPScore

30.5

DDS

PScore

DiffEdit

30.0

CLIPS

19.9

DiffEdit

29.5 -

19.8 -

Ours (10.0,5.0)

Ours(10.0,6.0)

29.0 -

19.7 -

EDICT

28.5

SDEdit

InstructPix2Pix

SDEdit

25

30

35

40

45

50

55

20

25

30

35

40

45

50

55

60

LPIPS

LPIPS

Results on ImageNet dataset. Figure 3 shows that our method performs well on images even with complex backgrounds. For instance, when editing "custard apple" into "lemon", our generated image successfully preserves the intricate details of the tree branches. DiffEdit [11] generates unnatural images with artifacts. DiffuseIT [39] generates unnatural images, while DDIB [40] can hardly maintain the content of the source image. Figure 5(a) shows that our method outperforms other methods by achieving a better trade-off between CLIPScore and LPIPS value.

Results on Imagen dataset. We present our qualitative and quantitative results in Figures 4 and 5(b), respectively. Figure 4 shows visual results including background replacement and object properties modification. We can see that our method can generate images with better visual quality compared with the other methods. For instance, our results can successfully preserve the foreground while replacing the "beach" in the background with "mountain", or vice versa.

Results on COCO-S dataset. From Figure 5(c), when a = 10 . 0 and b = 6 . 0 , our method achieves the highest CLIPScore 31.45, and the smallest LPIPS value 23.43 compared with all other methods. InstructPix2Pix [41] is able to obtain CLIPScore comparable to that of ours, but the LPIPS value of InstructPix2Pix [41] is worse. We provide qualitative comparison in Appendix B.2.

Results on DreamBooth dataset. Figure 5(d) shows the quantitative results on DreamBooth dataset. As can be observed, our method outperforms compared methods by achieving a better trade-off between CLIPScore and LPIPS value.

DDS [44] achieves the best CLIPScore of 20.17 with a worse LPIPS value of 33.26. Our method obtains CLIPScore comparable to that of DiffEdit with better LPIPS value. We provide qualitative comparison in Appendix B.2.

User study. We conduct two user studies on AFHQ dataset and the remaining datasets including ImageNet, Imagen, COCO-S, and DreamBooth. For each user study, 40 participants are provided with 30 randomly selected source images and the corresponding generated results of different methods. The generated images of ours and the other methods are displayed randomly in order. Participants are suggested to select the image that better applies the requested edit while preserving most of the original image details. The percentages of votes for

our method and the other method on different datasets are shown in Tables 2 and 3 respectively, demonstrating that the participants exhibit a strong preference towards our method.

Effect of hyper-parameters a and b . As mentioned in Section 3.4, we transform the attention map A ( I ) into weighting matrix Γ ( I ) with hyper-parameters a and b . We control the initial time step of denoising each pixel by adjusting the hyperparameters a and b . In Table 4, we analyze the impact of hyper-parameters a and b on AFHQ dataset. The upper part of the table reports the results with varying a and fixed b = 5 . 0 , and the lower part of the table shows results with varying b and fixed a = 10 . 0 . With b held constant, a larger a results in higher level of noise to the image, leading to more accurate editing and obtaining smaller FID. A smaller a results in content preserving the original image, obtaining larger SSIM value. The behavior of b is opposite to a .

Comparison with hard weighting ma-

trix. We compare with the strategy used to produce Λ ( I ) by Λ ( I ) = I ( A ( I ) ≥ η ) , where η is a threshold chosen in { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } . The generated weighting matrix by this strategy is a 'hard' weighting matrix that only takes 0 or 1 entry. Our Algorithm 1 gradually increases the denoising region with the increase of the denoising steps in the diffusion. Each pixel begins to be denoised with the denoising time step according to its relevance to the editing task. This helps to avoid artifacts caused by a hard mask, as shown in Figure 7.

Source

cat

↓

dog

fig

↓

garden

↓

beach

Ours

Failure examples We show several failure examples in Figure 6 that were unsuccessfully edited. This could be because the computed weighting matrix is not accurate for determining the scale of noise to be added to each pixel in the source image.

5 Related Work

Image editing aims to modify a real image to generate the desired image, resulting in tasks including image translation [46], style transfer [47], inpainting [48], object modification [24], etc. We focus on image editing tasks that commonly require editing specific object/thing of an image while preserving the remaining parts of the image.

Source

Ours

0.9

0.7

0.5

0.3

0.1

-1

(z)v

-0

Diffusion models showcase remarkable results for image editing. SDEdit [13] employs the source noisy data as the starting point in the denoising process, and it explores the trade-off between realism and faithfulness by controlling the initialization of the denoising time. EGSDE [14] utilizes the energy functions trained on both source and target domains to guide the inference process. EGSDE uses the noisy data at 0 . 5 T time step as the starting point of denoising, where T denotes the total number of denoising time steps in DDPM. Compared with them, our derived sampling method incorporates an adaptive selection of the initial denoising time for each pixel during the denoising process. Methods of [12, 23, 24, 49] utilize text-conditioned diffusion models to fine-tune the text embedding of the object that needs to be preserved during editing using a few or a single image. DDS [44] utilizes delta scoring to provide effective gradients for editing. DiffEdit [11] uses the DDIM inversion method to obtain noisy data and automatically generates a binary mask to guide the denoising process. RePaint [50] tackles the inpainting task by taking the unmasked image region from the input image and the masked region from the DDPM generated image, using the hard 0-1 mask. Differently, we perform image editing by adding independent noise with different variances to different pixels, depending on a weighting coefficient matrix that contains soft weights. Pixels with less added noise will better preserve the content of the source image.

6 Conclusion, Limitations and Societal Impact

In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing tasks. The NGDM adds independent Gaussian noises with varying variances to different image pixels. To avoid training score model for NGDM, we rectify NGDM into an isotropic Gaussian diffusion model and design a data sampling method for NGDM by using a pre-trained isotropic Gaussian diffusion model to generate images. We demonstrate that NGDM better trade-off the balance of realism and faithfulness than the state-of-the-art methods for image editing tasks.

A limitation of our method could be that incorrect weighting matrix may lead to the failure of the method. Moreover, our method relies on a pre-trained diffusion model. Artifacts are produced when the edit involves generation failure cases of the underlying model. In future work, we will design a better way to calculate the weighting matrix more precisely and efficiently. And we will explore the application of our method in downstream tasks, such as domain adaptation.

In our experiments, all the considered datasets are open-sourced and publicly available. Our work aims to manipulate images with minimum effort. However, this method might be misused by faking images. We will take care to exploit the method to avoid the potential negative social impact and we will help research in identifying and preventing malicious editing.

References

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurIPS , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS , 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021.

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR , 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV , 2021.

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. In ICLR , 2023.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR , 2023.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. In ICLR , 2022.

Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. In NeurIPS , 2022.

Xiang Gu, Liwei Yang, Jian Sun, and Zongben Xu. Optimal transport-guided conditional score-based diffusion model. In NeurIPS , 2023.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurIPS , 2022.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR , 2022.

Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In ICLR , 2023.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. In ICLR , 2022.

Vikram Voleti, Christopher Pal, and Adam Oberman. Score-based denoising diffusion with non-isotropic gaussian noise models. In NeurIPS , 2022.

Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392 , 2022.

Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05442 , 2022.

Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In CVPR , 2023.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR .

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput , 23(7):1661-1674, 2011.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR .

Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Flexit: Towards flexible semantic image translation. In CVPR , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Hexiang Hu, Ishan Misra, and Laurens Van Der Maaten. Evaluating text-to-image matching using binary image selection (bison). In ICCV Workshops , 2019.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. IP , 13(4):600-612, 2004.

Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In ECCV , 2020.

Shikun Sun, Longhui Wei, Junliang Xing, Jia Jia, and Qi Tian. Sddm: Score-decomposed diffusion models on manifolds for unpaired image-to-image translation. In ICML , 2023.

Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. In ICLR , 2023.

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR , 2022.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV , 2023.

Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In CVPR , 2023.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , 2017.

Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE Trans Vis Comput Graph , 26(11):3365-3385, 2019.

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR , 2018.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR , 2023.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGaussian diffusion is a common concept used in image processing, physics, and machine learning. It refers to a process where values (like the pixels in an image) are \"smoothed out\" or diffused according to a Gaussian function (also known as a normal distribution or bell curve), which provides a natural and mathematically convenient model for this diffusion process.  In the context of Gaussian diffusion, the term \"isotropic\" means that the diffusion is uniform in all directions. In contrast, \"non-isotropic\" Gaussian diffusion means that the diffusion is not uniform in all directions. The amount a pixel gets diffused might depend on its direction. \n\nThe authors propose a method to use a pre-trained Isotropic Gaussian Diffusion Model (IGDM) for sampling in the context of a Non-isotropic Gaussian Diffusion Model (NGDM). In the first step, they define the NGDM with added independent non-isotropic Gaussian noise. This suggests that they are introducing a type of Gaussian noise that's directionally dependent, in line with the concept of non-isotropic diffusion. Then, they detail how to implement the NGDM using the pre-trained IGDM. This process involves 'rectifying' the spatially different times of the noise and denoise procedures in the NGDM. It seems they're addressing the inherent differences between isotropic and non-isotropic models and adjusting the processes accordingly. Finally, they present a data sampling algorithm for the proposed NGDM that uses the pre-trained IGDM. This suggests that they're leveraging the pre-trained model's capabilities in the new context of non-isotropic diffusion.\n\nSTRENGTHS:\n1. Overall, they're taking an existing model trained on isotropic diffusion, adapting it for non-isotropic diffusion, and presenting a new algorithm for sampling data within this context.\n\n2. The authors show through experiments that their proposed method to be competitive.\n\nWEAKNESSES:\n1. The authors discussed the settings of the parameters \\alpha and \\beta but I do not see a strong link to the quality of the generated images.\n\n2. From the generated images shown in Figures 2-4, 6, I cannot really say that the proposed method outperforms the other consistently.  There is simply no \"wow\" effect.  Figure 6 shows that the color of the head also changed, not described in \"black leather jacket\".\n\nQUESTIONS:\n1. Please compare computational cost of all involved methods resented in Section 4.\n2. Please show a couple of examples where NGDM performs worse, and give the reasons to explain why.\n\nLIMITATIONS:\nIt is simply impossible to thoroughly \"prove\" a method is better by just displaying some selected results.  A much more robust and comprehensive empirical study followed by why the proposed NGDM works better would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel Non-isotropic Gaussian Diffusion Model (NGDM) for image-to-image translation and image editing tasks. The central idea of NGDM is to add independent Gaussian noises with different variances to different pixels, thus achieving controllable image translation and editing based on the amount of noise variance added to each pixel. Unlike prior models, the NGDM doesn't need specific training; instead, it rectifies into an isotropic Gaussian diffusion model where different pixels have varying total forward diffusion time.\n\nTo generate images from the diffused ones, the authors propose a sampling method that initiates denoising at different times for different pixels using a pre-trained isotropic Gaussian diffusion model. This process allows for the preservation of certain parts of the image while editing or translating others, thereby providing flexibility in performing such tasks.\n\nThe paper demonstrates the effectiveness of NGDM through experiments on three datasets encompassing real and synthetic images. The results show that NGDM outperforms state-of-the-art score-based diffusion models (SBDMs) in terms of FID and SSIM metrics for Cat → Dog image translation task. Similar superior performance was also observed in text-guided image translation and image editing tasks.\n\nIn summary, the main contributions of the paper are:\n1. The proposal of a novel Non-isotropic Gaussian Diffusion Model (NGDM) for image-to-image translation and image editing.\n2. Demonstration of how to rectify the NGDM into an isotropic Gaussian diffusion model, which allows leveraging a pre-trained isotropic model for denoising and image generation.\n3. A new sampling method for generating images by starting denoising at different times for different pixels.\n4. Validation of the proposed model against state-of-the-art SBDMs across several tasks and datasets, showing improved performance.\n5. Exploration of the trade-off between image fidelity and alignment with desired translation/editing targets, providing an avenue for controlling this balance with varying hyperparameters.\n\nSTRENGTHS:\n**Originality:** The proposed Non-isotropic Gaussian Diffusion Model (NGDM) demonstrates a high level of originality. It creatively deviates from the standard practice of isotropic diffusion by adding independent Gaussian noises with different variances to different pixels, enabling more controllable image translation and editing. This concept is novel and adds an interesting dimension to image generation models.\n\n**Quality:** The paper is of high quality. The authors provide a comprehensive explanation of the model, including detailed descriptions of the methodology. They describe the process of rectifying NGDM into an isotropic Gaussian diffusion model, which allows leveraging a pre-trained model for denoising and image generation. The experimental results on multiple datasets convincingly support their claims, showing that their method outperforms state-of-the-art score-based diffusion models in several tasks.\n\n**Clarity:** The paper is well-structured and clear. The authors do an excellent job explaining complex concepts in an understandable manner, making the paper accessible to readers with varying levels of familiarity with the topic. Diagrams and visual aids would further enhance the clarity of the paper.\n\n**Significance:** This work holds significant potential for advancing the field of image-to-image translation and image editing. The method provides a new way to control the translation/editing process by varying the noise added to different pixels, which could be invaluable in numerous applications. Moreover, this model's superior performance across multiple tasks and datasets shows promise for practical use in real-world scenarios. By demonstrating how to leverage existing isotropic Gaussian diffusion models within the NGDM framework, the authors also open up possibilities for future research in this area.\n\nWEAKNESSES:\nWhile the paper presents a novel and promising approach in the Non-isotropic Gaussian Diffusion Model (NGDM) for image-to-image translation and image editing tasks, there are several areas that could be improved:\n\n1. **Contextual Understanding:** The authors might consider providing more information on how their work compares to other existing methods. While they mention a few specific models and techniques, a more extensive literature review and comparison would be beneficial. This would allow readers to better understand the novelty and advantages of NGDM.\n\n2. **Experimentation:** The experiments conducted show promising results; however, the testing could be expanded. The authors only use three datasets for validation. More diverse datasets could help provide a more comprehensive performance evaluation of the method across different scenarios. Also, it would be helpful if the authors could include comparative visual results along with the quantitative metrics.\n\n3. **Model Interpretability:** While the model's performance is impressive, its interpretability seems not addressed. It might be challenging to understand why specific noise variances are assigned to certain pixels during the diffusion process. Providing some insights or analysis into this aspect may make the model more understandable.\n\n4. **Potential Limitations and Pitfalls:** The paper lacks a discussion about potential limitations and considerations when using the proposed model. Addressing possible issues such as computational cost, scalability, and any conditions under which the model might fail or perform suboptimally would provide a more balanced and realistic view of the proposed method. \n\n5. **Societal Impact:** As with any machine learning model, it's crucial to consider the ethical implications and potential misuse cases. Given the model's capacity for image manipulation, discussions around data privacy and consent, as well as potential ways the technology could be misused, should be included.\n   \nThese suggestions should help to strengthen the paper and broaden the understanding and applicability of the proposed NGDM.\n\nQUESTIONS:\nHere are a few questions and suggestions that would benefit from further clarification by the authors:\n\n1. **Noise Variance Distribution:** How is the variance of the Gaussian noise chosen for each pixel? Is there any strategy or algorithmic process behind this, or is it random? Knowing more about this could help understand how the NGDM manages to preserve certain parts of an image while translating/editing others.\n\n2. **Comparison with Other Models:** Could you provide a more detailed comparison between NGDM and other state-of-the-art models? It would be beneficial to include both qualitative (visual comparisons) and quantitative (additional metrics) results where possible.\n\n3. **Performance on Different Scenarios:** How does the model perform when applied to different tasks, especially those that have not been covered in the experimental section? Understanding its versatility and limitations across various scenarios will paint a fuller picture of NGDM's applicability.\n\n4. **Computational Efficiency:** Could you elaborate on the computational efficiency of your model? Specifically, how does the time complexity of the proposed method compare to other isotropic diffusion models? Does adding noise with different variances to different pixels significantly increase computational cost?\n\n5. **Ethical Considerations:** As your model allows substantial manipulation of images, what ethical considerations should be taken into account? Discussions around potential misuse, data privacy, and consent would add value.\n\n6. **Potential Improvements:** Lastly, can you suggest potential avenues for further improving the performance of NGDM? This might include parameter tuning, integrating with other models, or even extending the model to other domains beyond image translation and editing.\n\nLooking forward to your responses to these queries and suggestions, which I believe will provide a clearer understanding of your work and its implications.\n\nLIMITATIONS:\nBased on the provided information, it appears that the authors have not thoroughly addressed the potential limitations and negative societal impacts of their work.\n\nHere are a few suggestions for improvement:\n\n**Limitations:**\n\n1. **Computational Efficiency:** The authors should address the computational efficiency of their proposed model. Adding noise with different variances to different pixels might increase computational cost, which could be a limitation in some scenarios.\n   \n2. **Generalizability:** The paper does not discuss the performance of the model when applied to other tasks or datasets beyond those tested. Addressing this would give a clearer picture of the model's versatility and robustness.\n   \n3. **Model Interpretability:** The interpretability of the NGDM also seems unclear. Understanding why specific noise variances are assigned to certain pixels during the diffusion process could be complex. Discussing these aspects would help readers better understand and apply the model.\n\n**Societal Impacts:**\n\n1. **Ethical Considerations:** As the proposed model allows substantial manipulation of images, it is important to consider the ethical implications. The authors should provide a discussion about potential misuse, data privacy, and consent.\n\n2. **Potential Misuse:** With advanced image editing and translation capabilities, there may be potential for misuse of the technology, such as deepfake creation or unauthorized alteration of images. The authors should address these concerns and possible measures to prevent misuse.\n\nDiscussing these potential limitations and societal impacts would create a more balanced view of NGDM and help prepare users for any challenges they might face when applying the model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
The proposed model use differentiated reverse sampling strategy for image editing and translation.

STRENGTHS:
The proposed model use differentiated reverse sampling strategy for image editing and translation.

WEAKNESSES:
1. Please include the user study results on evaluating the natural image editing for qualitative evaluation.

2. The method lacks novelty. Using different starting point with masked and non-masked region is just a variation of DiffEdit approach. Please elaborate the difference between baseline models.

QUESTIONS:
See the weakness part.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a Non-isotropic Gaussian Diffusion framework, which they used for Image-to-Image translation and editing, which apparently are basically “soft inpainting” tasks. The authors crafted a non-uniform version of regular gaussian diffusion but ultimately used some knowledge from it to drive a regular gaussian diffusion model. The method looks very similar to a naive inpaiting method, with some notable differences.\n\nSTRENGTHS:\nThe paper shows a framework for image-editing, which is an increasingly important task for practical deployment. The authors also justified their method with some theretical backing, which is good.\n\nSome good quantitative and qualitative results are shown.\n\nWEAKNESSES:\nDespite the method being overall sound, I feel it is explained with more complication than necessary. The crux of the method is relatively simple than it looks.\n\nEffectively, what the method is doing is the following: It creates a “soft mask” (as shown in Fig.1 of supplementary) $\\Lambda_k = [ \\lambda_k ]_k$ for a given image that dictates which pixel $k$ requires more change in order to accomplish the translation/editing task. A time endpoint $T_k$ is created for each pixel based on $\\lambda_k$ — which is higher for pixels that require relatively more change during the generative process. Starting the generative process at $t=T$, the generation is “faked” (use of Eq. 11) till every pixel hits it’s own $T_k$, after that, the original de-noiser is used to “fill the gaps”.\n\n- In light of the above explanation, it seems all the theoretical description about NGDM is sort of unnecessary and over-complicated. The NGDM isn’t really used in it’s true sense. All it does is figures out $T_k = \\xi_k(t = T)$ — the expression Eq. 8 isn’t really necessary apart from it’s value at $t=T$. Essentially, the NGDM part is solely used to establish “how long to fake the generation for each pixel”.\n- Carrying on the previous point, the expression of $\\xi_k(t=T)$ is also arbitrarily related to the soft mask values $\\lambda_k$. There are many possible ways to incorporate $\\Lambda(\\mathcal{I})$ into the forward SDE, opening design choices for Eq.6. Depending on how $\\Lambda(\\mathcal{I})$ is incorporated into the SDE, the expression in Eq.8 will change. In that sense, it’s not clear as to why this specific design was chosen. One can completely do away with the NGDM framework and use an arbitrary function that relates $T_k$ to $\\lambda_k$ — what’s wrong with that?\n- Section B.2 has a rather good explanation with the hard-soft weighting matrix comparison. This should’ve been in the main paper and the explanation should’ve been geared towards “in-paiting with soft wighting”.\n- The really important part of this framework seems to be computation of the attention map $\\mathcal{A}(\\mathcal{I})$, which isn’t really a contribution of this paper (section 3.4). How costly is that, in terms of computation and relative to the core editing part ?\n- Quantitave results are pretty limited, unlike qualitative ones.\n- What’s the difference between the editing and image-to-image translation task? How do you incorporate “cat” and “dog” class into the translation task? Isn’t it related to text-based editing itself?\n\nOverall, I like the method, but I think the paper should’ve been written in a very different way. I would like the authors to take inspiration from the RePaint [1] paper.\n\n[1] https://arxiv.org/pdf/2201.09865.pdf\n\nQUESTIONS:\nPlease see the weakness section for consolidated comments and questions.\n\nLIMITATIONS:\nNot much limitation is mentioned. Some failure case and their possible explanation is written in suppl. They should be in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a non-isotropic gaussian diffusion model, in contrast to current popular isotropic gaussian diffusion models. The paper also proposes new forward and reverse diffusion processes in accordance to the non-isotropic gaussian corruption framework that is proposed. The motivation behind using non-isotropic gaussian noise is that \"the diffusion model can generate more diverse novel content if adding noise with larger variance to the image while preserving the image content if adding smaller variance noise\". Experimental results on image editing show superior quantitative performance for the proposed method.\n\nSTRENGTHS:\n- The idea of using non-isotropic gaussians for diffusion corruptions seems novel, to the best of my knowledge.\n- The forward and backward processes make sense and seem to work well enough to achieve similar performance to vanilla isotropic models.\n- The design allows for use of isotropic models which makes this a flexible method that can be easily inserted in many applications without retraining the model, which is very positive.\n- The main strength seems to be that the method is able to insert a different level of detail in different parts of the image, which improves editing (since different parts of the image have to be edited in different ways to achieve strong results). For example some parts of the image should be largely preserved while others should be heavily structurally changed. Once can see this in Fig. 3 with the custard apple -> lemon example.\n\nWEAKNESSES:\n- It's hard to see, on average, a large perceptual difference and superiority of the method in the qualitative translation figures. Maybe pointing to critical regions would help, but on average it seems like other methods are not too bad. Also, how were the samples for figures selected?\n- A user study on a large amount of data would clarify how users perceive these changes.\n- The method does achieve low LPIPS for a large CLIP score, but DiffEdit is close in relative terms.\n- The paper focuses on image editing but the title does not include the term. Would be good to include to be more specific.\n\n\nSome related work that could potentially be included (not mine):\n\nBansal, Arpit, et al. \"Cold diffusion: Inverting arbitrary image transforms without noise.\" arXiv preprint arXiv:2208.09392 (2022).\n\nDaras, Giannis, et al. \"Soft diffusion: Score matching for general corruptions.\" arXiv preprint arXiv:2209.05442 (2022).\n\nQUESTIONS:\nI think the paper is well presented and I understood it. The experiments are well laid out. The question would be to directly try to rebut the weaknesses I claim, and I will take into account other reviews to see whether I am being too harsh with respect to the relevance of the effect size on experiments (and the breadth of tasks tackled).\n\nLIMITATIONS:\nI think the main limitation is the motivation and experiments. Although the method is interesting, and the theoretical motivation (different parts of the image need to be treated slightly differently) is compelling, the final results don't seem to support the motivation so strongly and are isolated to the editing task. Maybe tackling more tasks with such a general methodology could be compelling? Or a user study that shows that users prefer these images, with large effect size?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

comments: SUMMARY:
The authors proposed a Non-isotropic Gaussian Diffusion Model for the task of image-image translation and image editing. The NGDM is achieved by adding different noise variances to different image pixels so as to control the regions to edit. Experimental results have demonstrated the state-of-the-art quality of the proposed paper.

STRENGTHS:
- The paper proposed a practical method for image-image translation/editing by utilizing the off-the-shelf diffusion model. It seems like the proposed method is easy to re-implement and has demonstrated satisfactory results.

- The presentation of the paper is good.

WEAKNESSES:
- Why the proposed method would choose to implement the non-isotropic diffusion process by controlling each pixel's denoising steps? I can come up with one alternative quickly: at each iteration, we can add different levels of noise to different pixels (by also using the input-dependent weight matrix), and denoise as usual. It is a bit strange to set different denoise time steps for different pixels.

- I would like to see comparisons with the recent paper "Delta Denoising Score", which also targets the task of controllable image editing.

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-29