Constructing Non-isotropic Gaussian Diffusion Model Using Isotropic Gaussian Diffusion Model for Image Editing
Authors: Xi Yu
Xiang Gu
Haozhi Liu
Jian Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: score-based diffusion model
non-isotropic Gaussian diffusion model
image editing
TL;DR: We propose a non-isotropic Gaussian diffusion model (NGDM) for image editing. By rectifying the NGDM, we can generate image by NGDM based on the pre-trained isotropic Gaussian diffusion model, e.g., DDPM.

Abstract: Score-based diffusion models (SBDMs) have achieved state-of-the-art results in image generation. In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing, which requires editing the source image while preserving the image regions irrelevant to the editing task. We construct NGDM by adding independent Gaussian noises with different variances to different image pixels. Instead of specifically training the NGDM, we rectify the NGDM into an isotropic Gaussian diffusion model with different pixels having different total forward diffusion time. We propose to reverse the diffusion by designing a sampling method that starts at different time for different pixels for denoising to generate images using the pre-trained isotropic Gaussian diffusion model. Experimental results show that NGDM achieves state-of-the-art performance for image editing tasks, considering the trade-off between the fidelity to the source image and alignment with the desired editing target.

Constructing Non-isotropic Gaussian Diffusion Model Using Isotropic Gaussian Diffusion Model for Image Editing
Xi Yu', Xiang Gu', Haozhi Liu, Jian Sun (><)
School of Mathematics and Statistics, Xi'an an Jiaotong University, Xi'an an, China
(ericayu, xi xianggu, liuhzh] fsstu.jttt wwt.iixxtuu........... edw.eddu.com (jiansun) fxjtu. www.eduuccc
Abstract
Score-based diffusion models (SBDMs) have achieved state-of-the-art results in image generation. In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing, which requires editing the source image while preserving the image regions irrelevant to the editing task. We construct NGDM by adding independent Gaussian noises with different variances to different image pixels. Instead of specifically training the NGDM, we rectify the NGDM into an isotropic Gaussian diffusion model with different pixels having different total for- ward diffusion time. We propose to reverse the diffusion by designing sampling method that starts different time for different pixels for denoising generate im- ages using the pre-trained isotropic Gaussian diffusion model. Experimental results show that NGDM achieves state-of-the-ar performance for image editing tasks, considering the trade-off between the fidelity to the source image and alignment with the desired editing target.
1 Introduction
Score-based diffusion models (SBDMs) [1-6] demonstrate state-of-the-ar performance on image synthesis quality and sample diversity. SBDMs are widely applied to applications such as text-to- image synthesis [7-9], image editing [10-15], deblurring [16, 17], etc. SBDMs consist of forward diffusion stage that adds random noise to data and reverse stage that generates desired data from noise. The introduced noise in the forward process is commonly isotropic Gaussian noise [1, 6]. i.e., independently and identically distributed noise normal distribution.
Non-isotropic diffusion model, by adding non-isotropic noises in the forward diffusion process has been investigated in [18-22]. The blurring diffusion model in [18] adds blur and noise samples which is Gaussian diffusion process with non-isotropic noise in the frequency space. [19] employs auxiliary velocity variables augment the data variables Hamiltonian dynamics, and performs the diffusion process in this expanded joint space by adding different noises to the auxiliary and the data variables. [20] formulates the diffusion model using non-isotropic noise with positive semi-definite covariance matrix, and carries out a comparative analysis of the non-isotropic and isotropic diffusion models. These works have shown better performance for data generation.
This paper focuses on image editing tasks that commonly require editing specific object/thing of an image while preserving the remaining parts of the image. For image editing, [11] produces a mask that allows the preservation of context while editing the remaining part. [23-25] use the learned text embedding for the object that needs to be preserved the image ensure the object unchanged
during editing. It is empirically known that the diffusion model can generate more diverse novel
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

content if adding noise with larger variance to the image while preserving the image content if adding smaller variance noise [13]. Motivated by this, we employ a non-isotropic diffusion model to add noises with different variances to different image pixels, considering the degree to which the corresponding pixels should be edited/preserved.
Along with this idea, we present a Non-isotropic Gaussian Diffusiio Model (NGDM), utilizing an off-the-shelf isotropic diffusion model (e.g.. DDPM [1]) for achieving the data sampling in image editing. Specifically, given a source image, the proposed NGDM is with a diffusion process that adds independent Gaussian noises with different variances to different pixels, therefore the added noise is independent and non- non-isotropic over image pixels. We then employ an off-the- isotropic diffusion model execute the reverse denoising process for image editing. To achieve this goal, we rectify the NGDM into the isotropic Gaussian diffusion model where each pixel is added with the same amount of noise at each step but different pixels have varying total number of noise accumulation time steps. We subsequently devise a specific sampling method for NGDM that can generate images by using the pre-trained isotopic Gaussian diffusion model.
We demonstrate the effectiveness of NGDM in image editing tasks on five datasets including real and synthetic datasets. In the experiments for cats to dogs editing task on AFHQ dataset, our method achieves the best performance in the metric of FID and SSIM compared with the SoTA SBDMs- methods. For text-guided image editing, our method achieves better trade-off between CLIPScore and LPIPS value. Furthermore, NGDM allows for flexible trade-off with varying
2 Background: Score-based Diffusion Models
SBDMs [1, 2, 4-6] are family of generative models that learn the data distribution based on the Gaussian process. Two representative models are Denoising Diffusion Probabilistic Model (DDPM) [1] and Score Matching with Langevin Dynamics (SMLD) [5]. We discuss the details based on DDPM for the remainder of the main text for brevity.
Given the input data x() R.D, which represents sample from the data distribution Pdatas forward process produces the noisy x(t) indexed by a time variable E [0, 1] via
x(t) = vā(t)x(0) + v1 - (t))
(I)
where z(t) E I), for any t and at) = e ((s)ds controlling the noise schedule. B(s) Smin + s(3max Bmii) with 3min 0.1 and 3max 20 [1, 6]. This type of diffusion model is dubbed Isotropic Gaussian Diffusion Model (IGDM), since the added Gaussian noise z(t) is from the independently and identically distributed normal distribution.
DDPM in the framework of Stochastic Differential Equations (SDEs) [5] with variance preservation
dx(()= 1
(1)x(()) + (((()==-s(x(())l)) VB(L)ddw with initial value x())
where w is the standard Wiener process. The reverse process denoises the noisy sample x(T) starting from T using a reverse SDE
1 dx(t) = - 2 8(t)x(() - Vx) log (x())(((1)) dt + /B(t)dw with initial value x(T),
(3)
where w is standard Wiener process when time flows backward from T to 0. The score function
Vx log P: (x) is approximated by training a time-dependent model sæ (x()),, t) via score matching [6,
26]. For inference, the time of the differential equation is discretized as E {0, At, 2At, ... .T} with
At representing the sampling time interval. We can choose to utilize the reverse process of DDPM or the reverse process of DDIM for sampling. With 3e = (()) the iteration rule of DDPM [1] is
x(t) = V1- -1-Bin -Btar (x(t + At) + SI4 (x(t + At), t + At) + VitAsiz(t + At),
(4)
where z(1 + At) E IO, With a. = 1I (1 3.), the iteration rule of DDIM [27] is
x(1) = Vivl! x(1 + At) (1 - 0e+ Ar) So (x(1 (t + +
Vor+ai
(5)
- /(1 (1- - āj 02(t+ + - (11(1--+r)) Sg (x(t+ + At), + At) + +o(t + +t))((++ + At).
2

---

NGDM
y(O)
Ti
Forward 1>
Tb Vo
Tv Reverse T
NGDM
ranelated
image
image
IGDM
Transiated
imagee
Figure 1: The overview of our NGDM. We rectify the non-isotropic diffusion model into isotropic model with different total time steps (e.g.. Tı.". ... _Tb) for different pixels. Based on this rectification, the input data y0) is firstly added isotropic noises until T time steps. Then in the reverse denoising process, to apply the pre-trained isotropic Gaussian diffusion model, we construct the noisy image to be denoised in each time step following Eq. (12). The red arrow in the figure indicates the pixel replacement operation in Eq. (12) when T3 I L S T for the k-th pixel at a denoising time step L.
3 Method
In this section, we present framework for utilizing the pre-trained Isotropic Gaussian Diffusion Model (IGDM) to achieve the sampling process of Non-isotropic Gaussian Diffusion Model (NGDM), In the following, first, we define the NGDM with added independent non-isotropic Gaussian noise. Then, we implement NGDM by IGDM through rectifying the spatially different time of noising and denoising procedures in NGDM and present our proposed data sampling algorithm for the proposed NGDM using the pre-trained IGDM.
3.1 Non-isotropic Gaussian Diffusion Model
In this work, we focus on the Non-isotropic Gaussian Diffusion Model (NGDM) by adding the non- isotropic Gaussian noise in the input data (0) IR'' and y(0) ~ Pdata, and the noises associated with different pixels are independent. The forward SDE NGDM is
1 3() + (T)() with initial value y(0),
dy
where I E RD is the source data, ATT) RDD - DxD is the weighting coefficient ma- trix, defined as diagonal matrix A(T) = diag (Aı, ,.) with 0 < A < 1 scaling the Gaussian noise level added the k--th pixel. Note that the transition kernel Por (t))
N (t) (0)e 1/2 ((s) I - (s)A(()d is an independent Gaussian distribution.
3.2 Rectify the Non-isotropic Gaussian Diffusion Model
With the added independent noise, we next discuss the forward SDE for NGDM in scalar form for each pixel h. Given the initial y²(0) (0) denoting the value of pixel k in y0), the forward SDE of the k-th pixel can be presented by
dy 3(1)Aay + (B(())ddw with initial value y* (0)),
(7)
where is one-dimensional Wiener process.
We present Theorem to illustrate the connection between the NGDM defined in Section 3.1 and the IGDM defined in Section 2 the pixel level. Beforchand, we introduce the following lemma.

---

Lemma 1. Let ((s) = Pmia + s(Smax - Pussu) with Bmax > Buin > 0. Then, for each A E [0, and t E [0, 1), there exists unique time [0, 1] (denoted by T (t) such that So 3(s) So B(s)ds and 3(1)2adl= = ((T) with the following form
(x(t) -Buinn v / + 2Bmmax Brmin) Pmintde + (Bmax 3miin 12
(8)
Binax
Pmin
The proof is in Appendix A. Based on the above Lemma, we can rectify the NGDM, which adds noise at each pixel with varying variance over the same time span, into an IGDM that adds noise at each pixel with the same noise variance but with different total diffusion time for different pixels. We introduce the following theorem to derive the differential equation as an IGDM.
Theorem 1. For a pixel indexed by k. X E [0, 1], and let = (t( with & (t) represented in Eq. (8).
With the same initial value y" (0), we have that the transition kernel at time t induced by Eq. (7) is equal to the transition kernel at time т induced by the following differential equation
1
dy = 3(T)y
dr + VB(r)dw
with initial value y (0)-
(9)
2
The total time of noising for Eq. (9) is T3 with TB = (T).
We provide the proof in Appendix A. Inspired by this, we rectify the reverse process in NGDM with different speeds of denoising across pixels to be the reverse process with consistent speed but different total time of denoising. We suggest rectifying the differential equation for the reverse process of pixel within the NGDM framework into the following form
dy - B(T))* ((T) log (y))* dr + V B(r)dw with initial value y (Tk), (10)
where w is a one-dimensional Wiener process when time flows backward from Ta to 0. Theorem I establishes the conclusion that the NGDM in Eq. (7) can be rectified to the IGDM in Eq. (9) but with different total diffusion time Ti for different pixel indexed by k, determined based on Eq. (8). This inspires to utilize the pre-trained IGDM to achieve the data sampling of NGDM for image editing. Subsequently, we present a method that adjusts the total time of noising and denoising for each pixel k to Th. enabling to use the pre-trained IGDM for data sampling. The corresponding sampling method is presented in Algorithm 1.
For image editing tasks, we use the source image I as (0) and generate noisy data (T) through the forward process. We generate the edited image (0) by denoising from (T)). Utilizing the forward noising process of IGDM, we add independent noise to each pixel k to obtain the noisy observation x (t) of discrete time t {0, At,... ,T with At representing the sampling time interval
x*(1) Va + VI- ar)"((),
(1))
where z(t) E NO, I). Next, with Hy + At), t + At) denotes the sampling procedure of DDPM and DDIM given in Eq. (4) and Eq. (5) of Section 2, the data sampling iteration utilizing the IGDM
model with initial value (T) = x (T) is defined as
(() (x)()
Th S otherwise.
(12)
(y(t+
This implies that we use the noisy observation x² (t) to represent y (t) at each step before Tg with t >> Tg. rather than the actual denoised result starting from time step T. Until time T2. we begin the denoising from (Tx)) for k-th pixel. In such a way, different pixels have different starting time steps (Tx for k-th pixel) for image denoising in the data sampling process.
3.3 Sampling Method in NGDM
Based on the above method, we further specify our sampling algorithm by utilizing pre-trained IGDM. We do not require the training of NGDM, but instead harness the power of pre-trained IGDM for data sampling. We generate the edited image with the source image I as a condition. We first add noise to the source image Z to T time steps as the starting point of denoising, and then use the method in Section 3.2 to rectify NGDM into IGDM to denoise the image using the pre-trained IGDM. We show the sampling algorithm of NGDM in the Algorithm 1.

---

Algorithm/ 1 Sampling Method of NGDM
Inputs:
The source image I, the weighting matrix (I), the score function sp, the time schedule (3(1))=oz the maximal time steps
1: Compute T (S,(T),... :DD(T) according to Eq. (8)
2: 2 ~ (0,1))
3: y vārI + /1 âTZ
# The starting point of denoising
4: for = 1o do
5: x vā,I /1 6₂
# Sample the noisy source image at time step t
6: M = I[t T))
7:
# 1 is the D-dimensional all-ones vector y M0x- (1-M) y
8:
*+H((y,))
9: end for
Output: Generated image y conditioned on the source image I
3.4 Design of Input-dependent Weighting Matrix
We construct the weighting matrix A(I) based on the method for designing mask using a text- conditioned diffusion model [9] in DiffEdit [11]. Specifically, given the source image I, the text description of the source image, and the target description Q that describes the desired target image after editing. Following DiffEdit [11], we add noise to the source image up to 0.5I' step and use the texts R and Q as the conditions respectively for denoising in the current time step to estimate the score values by using the score network Sg. We derive attention map A(I) based the absolute difference of the estimated scores. We use the above method to compute 10 estimated absolute noise differences by running 10 times with different random seeds, averaging and performing Gaussian smoothing on the averaged map. Finally, we normalize the values of the smoothed map to [0. 1] as the final attention map A()). The pixel with larger value in the attention map should be added with the noise with larger variance, to sufficiently edit the content of the pixel. The pixel with smaller value in the attention map should be added with noise having smaller variance preserve the content of pixel. We transform the attention map A(I) into the weighting matrix (I)) through a Sigmoid function by (T) T+exp(- (a))) where and b are the pprrmpaareestess for the transformation. We discuss the effect of hyper-parameters a and on the generation of images in Section 4.
Experiment
In this section, we apply the proposed NGDM to image editing tasks, presenting both qualitative and quantitative results.
4.1 Experimental Setup
Evaluation tasks. We perform experiments on five datasets. First, we experiment on AFHQ dataset [28] to edit cats into dogs. The source domain contains 500 images of cats. These images are resized to 256 x 256 resolution and subsequently edited into dogs. Second, we experiment on ImageNet dataset [29] to edit images from one class into another class based on text prompts, following the protocol of [30]. Third, we experiment on synthetic Imagen [31] dataset following DiffEdit [11]. We collect images generated by Imagen [31], along with their corresponding text prompts, and edit these images by altering portions of the text. Fourth, we experiment on COCO-S dataset and construct target prompts for editing from annotations provided by BISON [32]. We collect 1000 images and target prompts from the COCO [33] dataset to build the COCO-S dataset. We additionally consider DreamBooth dataset provided by [24], which contains 30 objects. Each object has 25 prompts and 4-6 images for editing. We edit each image using the provided 25 prompts, resulting in a total of 3950 edited images. More details are available in Appendix B.I.
Implementation details. We conduct experiments utilizing two types of pre-trained diffusion models. For the cats to dogs editing task on AFHQ dataset, we utilize the public pre-trained score- based diffusion model with the official code provided in ILVR [10]. This model operates directly in the image space. We set the denoising step N to 1000. We implement the remaining task based on the text-to-image latent diffusion model, i.c., Stable Diffusion [9]. This model was pre-trained on 512
5

---

x 512 images of LAION dataset [34] with a latent dimension of 64 x 64. We use 50 steps in DDIM sampling with a fixed noise schedule. For paper-parameters and b, we them to 10.0 and 5.0 respectively obtain all qualitative comparison results presented in this paper. We conduct additional analysis to investigate the effect of different values of hyper-parameteen in the experimental results. Unless otherwise stated, we use the default parameters of the diffusion model during inference.
Source
Ours
RESSaR S [44]
SpEEa [e1]
ILVR [10]
Figure Qualitative comparison on AFHQ dataset.
4.2 Results and Analysis
Results on AFHQ dataset. We report the widely-used Frechet Inception Distance (FID) [35] for quantifying realism and SSIM [36] for quantifying faithfulness. The quantitative comparisons and qualitative results are presented in Table 1 and Figure respectively. The images generated by our method better preserves the con-
textural structure (e.g., pose) of cats while yielding realistic dog images. For example, Figure 2 shows that for im- ages in columns 2-5 with complex back- grounds, our method can accurately keep the backgrounds unchanged while edit- ing cats into dogs. Other methods ei- ther blur the backgrounds or fail to main- tain the source image backgrounds cor- rectly. This observation is further sup- ported by the quantitative results in Ta- ble 1, where our method achieves the best results of FID and SSIM among the compared SBDM-based methods.
Table 1:
Quantitative comparison on AFHQ dataset. All results are reported by repeating experiments times.
Method
FID
SSIM
StarGAN v2 [28]
54.88
1.01
0.27 0.003
CUT [37]
76.21
0.601
ILVR [10]
74.37
1.55
0.363
0.001
SDEdit [13]
74.17 + 1.01
0.423 + 0.001
EGSDE [14]
65.82 0.77
0.415
0.001
SDDM [38]
62.29 + 0.63
0.422 + 0.001
Ours
61.39
0.27
0.478 0.001

---

Source
Oystercatcher
Broccoli +
Convertible
Custard apple
Bolete
Kite Bald eagle
Flamingo
Butternut squash
Ambulance
Lemon
Coral fungus
Ours
DifiEEdit
DiffuseIT
DDIB
SDEdit
Figure Qualitative comparison on ImageNet dataset.
Source
beach
fuzzy panda
cat
mountain
sunglasses hat
skateboarding
+
+
mountain
raccoon
Shiba Inu dog
beach
riding a bike
Ours
DiffEdit
SINE
SDEdit
Figure 4: Qualitative comparison on Imagen dataset.
7

---

30.5
8Ours [10 5.0
Diffuseff
@50cat
16.0
intrichh2Pis
25.8
SDEdit
@Ours (10.0,
CLi5Score
6,0)
@lmstrucPP:2PPx
35.4
29.0
eDfffed
35.2
DiffEoit
28.5
28.0
eDSS
34.8
@SINE EDICCT
@EDICT
eDDS
27.4
LPIPS
LPIPS
(a)
(b)
Ours (10.0,
8005
6.0)
20.1
@EDICT
30.5
30.0
19.9
aDrfEdit
24.5
19.8
(10.0, 5.0)
Ours [10.0, 6.01
29.0
39.7
28.5
InstructPi22Pxee
25
30
40
35
25
30
40
45
50
55
60
LPIPS
LPIP
(c)
(d)
Figure 5: Quantitative comparison on ImageNet (a), Imagen (b), COCO-S (c). and DreamBooth (d) datasets. We report LPIPS distance [42] measuring image fidelity and CLIPScore [43] for text alignment. A higher CLIPScore denotes better aligmment with the text, while lower LPIPS value suggests higher fidelity to the input image. We report our results respectively using the default parameters 10.0, b = 5.0, and the parameters = 10.0, b 6.0.
Results on ImageNet dataset. Figure 3 shows that our method performs well on images even with complex backgrounds. For instance, when editing "custard apple" into "lemon", our generated image successfully preserves the intricate details of the tree branches. DiffEdit [11] generates unnatural images with artifacts. DiffuseIT [39] generates unnatural images, while DDIB [40] can hardly maintain the content of the source image. Figure 5(a) shows that our method outperforms other methods by achieving a better trade-off between CLIPScore and LPIPS value.
Results on Imagen dataset. We present our qualitative and quantitative results in Figures 4 and 5(b), respectively. Figure 4 shows visual results including background replacement and object properties modification. We can see that our method can generate images with better visual quality compared with the other methods. For instance, our results can successfully preserve the foreground while replacing the "beach" in the background with "mountain" or vice versa.
Results on COCO-S dataset. From Figure 5(c), when a = 10.0 and = 6.0, our method achieves the highest CLIPScore 31.45, and the smallest LPIPS value 23.43 compared with all other methods. InstructPix2Pix [41] is able to obtain CLIPScore comparable to that of ours, but the LPIPS value of InstructPix2Pix [41] worse. We provide qualitative comparison in Appendix B.2.
Results on DreamBooth dataset. Figure 5(d) shows the quantitative results on DreamBooth dataset, As can be observed, our method outperforms compared methods by achieving better trade-off between CLIPScore and LPIPS value. DDS [44] achieves the best CLIPScore of 20.17 with worse LPIPS value of 33.26. Our method obtains CLIPScore comparable to that of DiffEdii with bet- ter L.IPS value. We provide qualitative comparison in Appendix B.2.
Table 2: User study results AFHQ dataset.
ILVR [10]
SDEdit [13]
EGSDE [14]
Ours
11.5%
10.5%
12.5%
65.5%

---

Table 3: User study results on ImageNet, Imagen, COCO-S, and DreamBooth datasets.
SDEdit [13] DiffEdit [11]
DDS [44] EDICT [45]
InstructPix2Pix [41] Ours
4.5%
10.0%
3.0%
4.5%
6.0%
72.0%
User study. We conduct two user stud- ies on AFHQ dataset and the remain- ing datasets including ImageNet, Ima- gen, COCO-S, and DreamBooth. For each user study, 40 participants are pro- vided with 30 randomly selected source images and the corresponding generated results of different methods. The gener- ated images of ours and the other meth- ods are displayed randomly in order. Par- ticipants are suggested to select the im- age that better applies the requested edit while preserving most of the original im- age details. The percentages of votes for our method and the other method different datasets are shown in Tables 2 and respectively, demonstrating that the participants exhibit a strong preference towards our method.
Table 4: The performance AFHQ dataset with varying
values of or while respectively fixing = 5.0 or
a 10.0.
(b 5.0)
6.0
8.0
12.0
14.0
FID
87.01
72,43
54.10
46.75
SSIM 1
0.556
0.513
0,449
0.425
(a = 10.0)
3.0
4.0
6.0
7.0
FID
42.34
50.35
74.58
88.49
SSIM
0.373
0.423
0.539
0.601
Effect of hyper- syper-parameters and b. As mentioned in Section 3.4, transform the attention map A(I) into weighting matrix (II) with hyper-parameters a and We control the initial time step of denoising each pixel by adjusting the hyperparameters and b. In Table 4, we analyze the impact of hyper-parametent a and on AFHQ dataset. The upper part of the table reports the results with varying and fixed b = 5.0, and the lower part of the table shows results with varying b and fixed 10.0. With b held constant, a larger a results in higher level of noise to the image, leading to more accurate editing and obtaining smaller FID. A smaller results in content preserving the original image, obtaining larger SSIM value. The behavior of b opposite to a.
Comparison with hard weighting ma- trix. We compare with the strategy used produce AII) by A(I) I((A(I) 2 1T). where 77 a threshold chosen in (0.1, 0.3, 0.5, 0.7, 0.9). The generated weighting matrix by this strat- egy is "hard" weighting matrix that only takes 0 or I entry. Our Algorithm gradually increases the denoising region with the increase of the denoising steps in the diffusion. Each pixel begins be denoised with the denoising time step according its relevance to the editing task. This helps to avoid artifacts caused by a hard mask as shown Figure 7.
cat +
fig +
garden
dog
strawberry
beach
Ours
Figure 6: Failure examples. We show cases in which our method fails to generate high-quality edited results.
Failure examples We show several failure examples in Figure 6 that were unsuccessfully edited. This could be because the computed weighting matrix is not accurate for determining the scale of noise to be added to each pixel in the source image.
5 Related Work
Image editing aims to modify real image to generate the desired image, resulting in tasks including image translation [46], style transfer [47], inpainting [48], object modification [24], etc. We focus on image editing tasks that commonly require editing specific object/thing of an image while preserving the remaining parts of the image.

---

Source
Ours 0.9
0.7
0.5
0.3
0.1
(I)
Figure 7: Edited images and heatmaps with soft and hard weighting matrix. The images in the second column represent the results generated by our method and the heatmap below the image depicts the weighting matrix A (I) defined in Section 1 in the paper. The images in columns 3-7 represent the results generated using the hard weighting matrix with threshold value in (0.1, 0.3, 0.5, 0.7, 0.9). The heatmaps below the images represent the binary hard weighting matrix.
Diffusion models showcase remarkable results for image editing. SDEdit [13] employs the source noisy data the starting point in the denoising process, and it explores the trade-off between realism and faithfulness by controlling the initialization of the denoising time. EGSDE [14] utilizes the energy functions trained on both source and target domains to guide the inference process. EGSDE uses the noisy data 0.57 time step as the starting point of denoising, where denotes the total number of denoising time steps in DDPM. Compared with them, our derived sampling method incorporates an adaptive selection of the initial denoising time for each pixel during the denoising process. Methods of [12, 23, 24, 49] utilize text-conditioned diffusion models to fine-tune the text embedding the object that needs to be preserved during editing using a few or single image. DDS [44] utilizes delta scoring to provide effective gradients for editing. DiffEdit [11] uses the DDIM inversion method to obtain noisy data and automatically generates binary mask to guide the denoising process. RePaint [50] tackles the inpainting task by taking the unmasked image region from the input image and the masked region from the DDPM generated image, using the hard 0-1 mask. Differently, we perform image editing by adding independent noise with different variances to different pixels, depending on weighting coefficient matrix that contains soft weights. Pixels with less added noise will better preserve the content of the source image.
6 Conclusion, Limitations and Societal Impact
In this paper, we propose Non-isotropic Gaussian Diffusion Model (NGDM) for image editing tasks. The NGDM adds independent Gaussian noises with varying variances to different image pixels. To avoid training score model for NGDM, we rectify NGDM into an isotropic Gaussian diffusion model and design a data sampling method for NGDM by using pre-trained isotropic Gaussian diffusion model to generate images. We demonstrate that NGDM better trade-off the balance of realism and faithfulness than the state-of-the-un methods for image editing tasks.
limitation of our method could be that incorrect weighting matrix may lead the failure of the method. Moreover, our method relies on pre-trained diffusion model. Artifacts are produced when the edit involves generation failure cases of the underlying model. In future work, we will design better way to calculate the weighting matrix more precisely and efficiently. And we will explore the application of our method in downstream tasks, such as domain adaptation.
In our experiments, all the considered datasets are open-sourced and publicly available. Our work aims manipulate images with minimum effort. However, this method might be misused by faking images. We will take care to exploit the method to avoid the potential negative social impact and we will help research in identifying and preventing malicious editing.
7 Acknowledgement
This work was supported by National Key R&D Program Y0A YFA 1003002, and NSFC (12125104, U20B2075.11971377)
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[2] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurIPS, 2021.
[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML., 2021.
[4] Jascha Dickstein,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015.
[5] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
[6] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[7] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220..06125, 2022.
[9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[10] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
[11] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthicu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In ICLR, 2023.
[12] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
[13] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[14] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-gun guided stochastic differential equations. NeurlPS, 2022.
[15] Xiang Gu, Liwei Yang, Jian Sun, and Zongben Xu. Optimal transport- guided conditional score-based diffuusion model. In NeurlPS, 2023.
[16) Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NewrlPS, 2022.
[17] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR. 2022.
[18] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. ICLR, 2023.
[19] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. In ICLR, 2022.
[20] Vikram Voleti, Christopher Pal, and Adam Oberman. Score-based denoising diffusion with non-isotropic gaussian noise models. In NewrIPS, 2022.
[21] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint 2XX::22088.0939, 2022.

---

[22] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milan- far. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05444 044422. 2022.
[23] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. CVPR, 2023.
[24] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR.
[25] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
[26] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput, 23(7):1661-166 2011.
[27] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In
ICLR, 2021.
(28] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Won Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
[29] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR.
[30] Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Flexit: Towards flexible semantic image translation. In CVPR, 2022.
[31] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[32] Hexiang Hu, Ishan Misra, and Laurens Van Der Maaten. Evaluating text-to-image matching using binary image selection (bison). ICCV Workshops, 2019.
[33] Tsung-Yo Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[34] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.021 2021.
[35] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
[36] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity, IEEE Trans. IP, 1304)6600-612, 2004.
[37] Taesung Park, Alexei A Efros, Richard Zhang, and Yan Zhu. Contrastive learning for unpaired image-to-image translation. In ECCV, 2020.
[38] Shikun Sun, Longhui Wei, Junliang Xing, Jia Jia, and Qi Tian. Sddm: Score-decomposed diffusion models on manifolds for unpaired image-to-ine -ongaee translation. In ICML, 2023.
[39] Gihyun Kwon and Jong Chul Ye. ceffusion-based image translation using disentangled style and content representation. In ICLR, 2023.
[40] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. ICLR, 2022.
[41] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
12

---

[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
[44] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV, 2023.
[45] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In CVPR, 2023.
[46] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[47] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE Truns Vis Comput Graph, 26(1 1):3655-3385, 2019.
[48] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, 2018.
[49] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[50] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
13


Reviews:
[{"review_id": "yhCx7w1rbz", "reviewer": "Reviewer_xHHG", "summary": "The authors proposed a Non-isotropic Gaussian Diffusion Model for the task of image-image translation and image editing. The NGDM is achieved by adding different noise variances to different image pixels so as to control the regions to edit. Experimental results have demonstrated the state-of-the-art quality of the proposed paper.", "questions": "See above", "limitations": "See above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposed a practical method for image-image translation/editing by utilizing the off-the-shelf diffusion model. It seems like the proposed method is easy to re-implement and has demonstrated satisfactory results.\n\n- The presentation of the paper is good.", "weaknesses": "- Why the proposed method would choose to implement the non-isotropic diffusion process by controlling each pixel's denoising steps? I can come up with one alternative quickly: at each iteration, we can add different levels of noise to different pixels (by also using the input-dependent weight matrix), and denoise as usual. It is a bit strange to set different denoise time steps for different pixels.\n\n- I would like to see comparisons with the recent paper \"Delta Denoising Score\", which also targets the task of controllable image editing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The reason for implementing NGDM by controlling each pixel’s denoising steps.**\n\nIt is empirically known that the diffusion model can generate more diverse novel content if adding noise with a larger variance to the image while preserving the image information if adding smaller variance noise. Motivated by this, we employ a non-isotropic Gaussian diffusion model to add noises with different variances to different image pixels for controllable image editing. \n\nAs suggested by the reviewer, we can directly add different levels of noise to different pixels, and denoise as usual using pre-trained diffusion models. Actually, we have tried this strategy when working on this paper, but only produced noises rather than images. The main reason could be that the transition kernel at time $t$ of the isotropic Gaussian diffusion model is different from that of the non-isotropic Gaussian diffusion model, which is related to the weighting matrix that determines the variance. Therefore, the score under isotropic Gaussian diffusion model at time step $t$ does not match the score under the non-isotropic Gaussian diffusion model. The pre-trained isotropic Gaussian diffusion model will produce incorrect score predictions at time $t$ for non-isotropic Gaussian diffusion model. \n\nOne way is to retrain the non-isotropic Gaussian diffusion model. We instead use the off-the-shelf pre-trained isotropic Gaussian diffusion model for achieving the date sampling to avoid retraining. To implement this idea, we use Lemma 1 in the main paper to establish a relationship between the noise level in the non-isotropic Gaussian diffusion model and the time step $t$ in the isotropic Gaussian diffusion model. We prove in Theorem 1 that the transition kernel at time $t$ in the non-isotropic Gaussian diffusion model is equal to the transition kernel at time step $\\tau$ in the isotropic Gaussian diffusion model, where $\\tau$ depends on the noise level in the non-isotropic Gaussian diffusion model. Finally, since isotropic Gaussian diffusion model needs to accept the entire image as input, we propose a sampling algorithm of NGDM (Algorithm 1) to generate images using pre-trained isotropic Gaussian diffusion model.\n\n\n**Q2: Comparisons with \"Delta Denoising Score\".**\n\nDDS [Hertz A, et al., arXiv:2304.07090, 2023.] utilizes score distillation sampling mechanism for image editing. The authors optimize to produce the edited image given a text description by distilling the output of the score-based model on the reference source image-text pair. Differently, we aim to edit the image and leave the region unrelated to target prompt unchanged. We achieve this goal using a non-isotropic Gaussian diffusion model that adds independent noise with different variances to different pixels. We further rectify it into an isotropic Gaussian diffusion model with different pixels having different total forward diffusion times, for utilizing the pre-trained isotropic models for sampling. \n\nWe conduct experiments on the ImageNet dataset and Imagen dataset introduced in Section 4 of the paper. In the rebuttal, we additionally consider two datasets (COCO-S and DreamBooth Dataset [Ruiz N, et al., CVPR2023]) for image editing task. \n\nWe show qualitative visual results on COCO-S and Dreambooth datasets in Figure r1 of the uploaded one-page pdf. We report the CLIPScore and LPIPS metric in Table r1-1, the results of user study in Table r1-2, and the computational time and memory cost in Table r1-3. Table r1-1 shows that our method consistently achieves the best results on the four datasets. A larger CLIPScore denotes better alignment with the target text, while a smaller LPIPS value suggests higher fidelity to the source image. Compared with DDS, our method achieves a smaller LPIPS distance with a larger CLIPScore, which shows that our method can make smaller changes to the source image when editing. \n\nTable r1-1: CLIPScore ($\\uparrow$) and LPIPS ($\\downarrow$) on four datasets.\n|Method|Imagenet (CLIPScore)|Imagenet (LPIPS)|Imagen (CLIPScore)|Imagen (LPIPS)|COCO-S (CLIPScore)|COCO-S (LPIPS)|Dreambooth (CLIPScore)|Dreambooth (LPIPS)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|DDS|28.13|43.57|34.69|31.38|30.48|24.00|**20.17**|22.52|\n|Ours|**30.66**|**31.32**|**35.66**|**20.28**|**31.75**|**23.43**|19.81|**19.95**|\n\nWe conduct user study by providing 40 participants with 30 randomly selected source images and the corresponding generated results of different methods are displayed randomly. Participants were asked to choose the image that best achieves the requested edit while preserving most of the original image details. The percentage of votes for each method is shown in Table r1-2. The results demonstrate that the participants exhibit a strong preference for our method. \n\nTable r1-2: User study results.\n\n|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|4.5%|10.0%|3.0%|4.5%|6.0%|**72.0%**|\n\nWe test the computational efficiency on NVIDIA GeForce RTX3090. From Table r1-3, it can be seen that the running time of DDS is about 7.7 times that of ours and the memory occupied by DDS is also higher than ours.\n\nTable r1-3: Computational time and memory cost.\n\n|Method|DDS|Ours|\n|-|:-:|:-:|\n|Time per iteration (s)$\\downarrow$|46|**6**|\n|Memory (GB)$\\downarrow$|16.7|**6.7**|"}, {"review_id": "Bv9QgBXl9I", "reviewer": "Reviewer_Jr5M", "summary": "The paper proposes a non-isotropic gaussian diffusion model, in contrast to current popular isotropic gaussian diffusion models. The paper also proposes new forward and reverse diffusion processes in accordance to the non-isotropic gaussian corruption framework that is proposed. The motivation behind using non-isotropic gaussian noise is that \"the diffusion model can generate more diverse novel content if adding noise with larger variance to the image while preserving the image content if adding smaller variance noise\". Experimental results on image editing show superior quantitative performance for the proposed method.", "questions": "I think the paper is well presented and I understood it. The experiments are well laid out. The question would be to directly try to rebut the weaknesses I claim, and I will take into account other reviews to see whether I am being too harsh with respect to the relevance of the effect size on experiments (and the breadth of tasks tackled).", "limitations": "I think the main limitation is the motivation and experiments. Although the method is interesting, and the theoretical motivation (different parts of the image need to be treated slightly differently) is compelling, the final results don't seem to support the motivation so strongly and are isolated to the editing task. Maybe tackling more tasks with such a general methodology could be compelling? Or a user study that shows that users prefer these images, with large effect size?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using non-isotropic gaussians for diffusion corruptions seems novel, to the best of my knowledge.\n- The forward and backward processes make sense and seem to work well enough to achieve similar performance to vanilla isotropic models.\n- The design allows for use of isotropic models which makes this a flexible method that can be easily inserted in many applications without retraining the model, which is very positive.\n- The main strength seems to be that the method is able to insert a different level of detail in different parts of the image, which improves editing (since different parts of the image have to be edited in different ways to achieve strong results). For example some parts of the image should be largely preserved while others should be heavily structurally changed. Once can see this in Fig. 3 with the custard apple -> lemon example.", "weaknesses": "- It's hard to see, on average, a large perceptual difference and superiority of the method in the qualitative translation figures. Maybe pointing to critical regions would help, but on average it seems like other methods are not too bad. Also, how were the samples for figures selected?\n- A user study on a large amount of data would clarify how users perceive these changes.\n- The method does achieve low LPIPS for a large CLIP score, but DiffEdit is close in relative terms.\n- The paper focuses on image editing but the title does not include the term. Would be good to include to be more specific.\n\n\nSome related work that could potentially be included (not mine):\n\nBansal, Arpit, et al. \"Cold diffusion: Inverting arbitrary image transforms without noise.\" arXiv preprint arXiv:2208.09392 (2022).\n\nDaras, Giannis, et al. \"Soft diffusion: Score matching for general corruptions.\" arXiv preprint arXiv:2209.05442 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: On the performance improvement over baselines and the way to select visualization samples**\n\nWe tackle controllable image editing that modifies the image regions related to the editing task while leaving the other regions unchanged. As shown in Figure 2 in the paper, our method can better preserve the background, pose, etc., compared with other methods. For example, Figure 2 shows that for images in columns 2-5 with relatively complex backgrounds, our method can accurately keep the backgrounds of the source images unchanged while translating cats into dogs. Other methods either blur the backgrounds or fail to maintain the source image backgrounds correctly. In Figure 3, the source images in columns 2 and 4 have complex backgrounds, and our method can better preserve these details when editing images. Besides the metrics reported in the paper, we also conduct user study in the rebuttal, which shows an obvious performance improvement achieved by our method (please refer to Q2).\n\nThese displayed examples are randomly selected from each dataset, typically with source images having various poses and backgrounds or with various editing types (such as background replacement or object transformation). Due to space limit, we can only show a few examples, but overall these conclusions are the same in the other examples of these images in the datasets. We also randomly selected several failure examples in Figure 6 of Appendix.\n\n**Q2: A user study on a large amount of data.**\n\nWe conduct user study by providing 40 participants with 30 randomly selected source images and the corresponding generated results of different methods are displayed randomly. Participants were asked to choose the image that better applies the requested edit while preserving most of the original image details. The percentage of votes for each method is shown in Tables r2-1 and r2-2. The results demonstrate that the participants exhibit a strong preference for our method. \n\nTable r2-1: User study results of Cat $\\rightarrow$ Dog task.\n\n|ILVR|SDEdit|EGSDE|Ours|\n|:-:|:-:|:-:|:-:|\n|11.5%|10.5%|12.5%|**65.5%**|\n\nTable r2-2: User study results of the tasks on ImageNet, Imagen, COCO-S, and Dreambooth datasets.\n\n|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|4.5%|10.0%|3.0%|4.5%|6.0%|**72.0%**|\n\n**Q3: The method does achieve low LPIPS for a large CLIP score, but DiffEdit is close in relative terms.**\n\nDiffEdit is close to our method on CLIPScore and LPIPS metrics. However, from a qualitative comparison point of view as shown in Figure 3, our method can generate more natural images, while DiffEdit is prone to produce boundary artifacts caused by hard mask, resulting in an unsmooth combination of foreground and background. Please refer to the images in the first and fourth columns of Figure 3 for the examples. This may not be reflected by the CLIPScore and LPIPS. Note that the results of user study in Table r2-2 show that the participants prefer the results of our method.\n\n**Q4: Including \"image editing\" in title.**\n\nThanks for this suggestion, we will consider changing the title to \"Constructing Non-isotropic Gaussian Diffusion Model Using Isotropic Gaussian Diffusion Model for Image Editing\", as suggested.\n\n**Q5: Including the related works [Bansal A, et al., arXiv:2208.09392, 2022] and [Daras G, et al., arXiv:2209.05442, 2022.]**\n\nWe will include these references in the Introduction section. Cold Diffusion [Bansal A, et al., arXiv:2208.09392, 2022] is based on generalized diffusion models that were built on arbitrary image transformations like blurring, downsampling, etc. And a trained restoration network is created to perform denoising. Soft diffusion [Daras G, et al., arXiv:2209.05442, 2022.] is based on general linear corruption processes and it learns the diffusion model by training objective of Soft Score Matching for the linear corruption process. Differently, we consider non-isotropic Gaussian noise and utilize a pre-trained isotropic Gaussian diffusion model to achieve sampling without retraining. \n\n**Q6: More experimental tasks and user study to support motivation.**\n\nImage editing task aims at modifying source images under the target prompt guidance while leaving the regions that are unrelated to editing unchanged, to generate images that are as similar as possible to source image. It is empirically known that the diffusion model can generate more diverse novel content if adding noise with a larger variance to the image while preserving the image information if adding smaller variance noise. Motivated by this, we employ a non-isotropic diffusion model to add noises with different variances to different image pixels. We achieve controllable editing by adding different variance noises to different image pixels, with varying noise variance considering the degree to which the corresponding pixels should be edited/preserved. We will explain this motivation for using NGDM for controllable image editing in more detail in the Introduction section.\n\n**More tasks.** We add experiments on two additional tasks that are not covered in the experimental section, including local style transfer and gender transformation. The local style transfer task aims to transform the specified object in the image into another style without changing the structure, while preserving the information of the rest of the region. For example, transforming a \"real dog\" into a \"sculptural dog\", while keeping the background unchanged. Gender transformation aims to turn males into females while keeping the structure of the face unchanged. Due to space limit, we show a few examples in Figure r2 of the uploaded one-page pdf, and we will show more examples in Appendix B in the revised version. \n\n**User study.** As suggested, we report the results of user study in Tables r2-1 and r2-2. The results show that the participants exhibit a strong preference for our method."}, {"review_id": "sfEEfqpstk", "reviewer": "Reviewer_Cf2x", "summary": "The authors proposed a Non-isotropic Gaussian Diffusion framework, which they used for Image-to-Image translation and editing, which apparently are basically “soft inpainting” tasks. The authors crafted a non-uniform version of regular gaussian diffusion but ultimately used some knowledge from it to drive a regular gaussian diffusion model. The method looks very similar to a naive inpaiting method, with some notable differences.", "questions": "Please see the weakness section for consolidated comments and questions.", "limitations": "Not much limitation is mentioned. Some failure case and their possible explanation is written in suppl. They should be in the main paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper shows a framework for image-editing, which is an increasingly important task for practical deployment. The authors also justified their method with some theretical backing, which is good.\n\nSome good quantitative and qualitative results are shown.", "weaknesses": "Despite the method being overall sound, I feel it is explained with more complication than necessary. The crux of the method is relatively simple than it looks.\n\nEffectively, what the method is doing is the following: It creates a “soft mask” (as shown in Fig.1 of supplementary) $\\Lambda_k = [ \\lambda_k ]_k$ for a given image that dictates which pixel $k$ requires more change in order to accomplish the translation/editing task. A time endpoint $T_k$ is created for each pixel based on $\\lambda_k$ — which is higher for pixels that require relatively more change during the generative process. Starting the generative process at $t=T$, the generation is “faked” (use of Eq. 11) till every pixel hits it’s own $T_k$, after that, the original de-noiser is used to “fill the gaps”.\n\n- In light of the above explanation, it seems all the theoretical description about NGDM is sort of unnecessary and over-complicated. The NGDM isn’t really used in it’s true sense. All it does is figures out $T_k = \\xi_k(t = T)$ — the expression Eq. 8 isn’t really necessary apart from it’s value at $t=T$. Essentially, the NGDM part is solely used to establish “how long to fake the generation for each pixel”.\n- Carrying on the previous point, the expression of $\\xi_k(t=T)$ is also arbitrarily related to the soft mask values $\\lambda_k$. There are many possible ways to incorporate $\\Lambda(\\mathcal{I})$ into the forward SDE, opening design choices for Eq.6. Depending on how $\\Lambda(\\mathcal{I})$ is incorporated into the SDE, the expression in Eq.8 will change. In that sense, it’s not clear as to why this specific design was chosen. One can completely do away with the NGDM framework and use an arbitrary function that relates $T_k$ to $\\lambda_k$ — what’s wrong with that?\n- Section B.2 has a rather good explanation with the hard-soft weighting matrix comparison. This should’ve been in the main paper and the explanation should’ve been geared towards “in-paiting with soft wighting”.\n- The really important part of this framework seems to be computation of the attention map $\\mathcal{A}(\\mathcal{I})$, which isn’t really a contribution of this paper (section 3.4). How costly is that, in terms of computation and relative to the core editing part ?\n- Quantitave results are pretty limited, unlike qualitative ones.\n- What’s the difference between the editing and image-to-image translation task? How do you incorporate “cat” and “dog” class into the translation task? Isn’t it related to text-based editing itself?\n\nOverall, I like the method, but I think the paper should’ve been written in a very different way. I would like the authors to take inspiration from the RePaint [1] paper.\n\n[1] https://arxiv.org/pdf/2201.09865.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Necessity of theoretical description and Eqs. (6)&(8)**\n\nWe clarify the necessity of theoretical description and Eqs. (6,8) as follows.\n\nFirstly, our motivation is to achieve controllable image editing by adding noise with different variances to different pixels of the image. This motivates us to construct the non-isotropic Gaussian diffusion model (NGDM).\n\nSecondly, to avoid retraining, we propose to use the pre-trained isotropic Gaussian diffusion model (IGDM) for achieving the data sampling for NGDM. The key challenge is how to formulate the relation between NGDM and IGDM. Section 3.2 analyzes the relation between the variance of noise in NGDM and the time step in IGDM. The deduced result in Eq. (8) establishes the transformation from noise variance to time step.\n\nWith this theoretical foundation, we can first calculate the weighting matrix $\\mathbf{\\Lambda}(\\mathcal{I})$ based on the input image $\\mathcal{I}$, then derive the denoising time steps of each pixel based on Eq. (8). If we do away with NGDM and discard the Eq. (8), we have no theoretical guarantee for designing function that relates $T_k$ to $\\lambda_k$. Our theoretical analysis and Eq.(8) in Lemma 1 build the theoretical support for the selection of the function with theoretical interpretation.\n\n**Q2: Including an explanation of the hard-soft weighting matrix comparison in the main paper with an explanation geared towards \"in-painting with soft weighting\".**\n\nWe will include the hard-soft weighting matrix comparison in the main paper. Mask-guided image editing is similar to the inpainting task. Both tasks have the same problem that the boundary of mask is prone to be with incorrect artifacts. Our Algorithm 1 gradually increases the denoising region with the increase of the denoising steps in the diffusion. Each pixel begins to be denoised with the denoising time step according to its relevance to the editing task. This helps to generate natural avoiding artifacts caused by a hard mask. \n\n**Q3: Cost of computing the attention map.**\n\nWe conduct experiments on the single NVIDIA GeForce RTX3090. For an image with resolution of 512 × 512, the computation time of the attention map $\\mathbf{\\Lambda}(\\mathcal{I})$ takes about 2.2 seconds, and the computation time of the core editing part takes about 3.8 seconds. A complete edit to an image takes about 6 seconds in total. \n\n**Q4: Quantitative results are pretty limited, unlike qualitative ones.** \n\nWe add two natural datasets (COCO-S and DreamBooth Dataset [Ruiz N, et al., CVPR2023]) and SoTA methods DDS [Hertz A, et al., arXiv:2304.07090, 2023], InstructPix2Pix [Brooks T, et al., CVPR2023] and EDICT [Wallace B, et al., CVPR2023] for more comparison. The quantitative results are shown in Table r3-1. It can be seen that our method outperforms competing methods by achieving a better trade-off between CLIPScore and LPIPS value.\n\nTable r3-1: Quantitative comparison on COCO-S and Dreambooth dataset.\n\n|Method|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours ($a$=10.0,$b$=5.0)|Ours ($a$=10.0,$b$=6.0)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CLIPScore$\\uparrow$ (COCO-S dataset)|28.58|30.31|30.48|30.77|31.28|**31.75**|31.45|\n|LPIPS$\\downarrow$ (COCO-S dataset)|55.76|29.60|31.53|24.00|36.55|28.80|**23.43**|\n|CLIPScore$\\uparrow$ (Dreambooth dataset)|19.64|19.88|**20.17**|19.70|19.64|19.81|19.75|\n|LPIPS$\\downarrow$ (Dreambooth dataset)|59.82|27.47|33.26|22.52|40.50|24.86|**19.95**|\n\nWe conduct user study by providing 40 participants with 30 randomly selected source images and the corresponding generated results of different methods are displayed randomly. Participants were asked to choose the image that better applies the requested edit while preserving most of the original image details. The percentage of votes for each method is shown in Tables r2-1 and r2-2. The results demonstrate that the participants exhibit a strong preference to our method. \n\nTable r3-2: User study of Cat $\\rightarrow$ Dog task.\n\n|ILVR|SDEdit|EGSDE|Ours|\n|:-:|:-:|:-:|:-:|\n|11.5%|10.5%|12.5%|**65.5%**|\n\nTable r2-2: User study results of the tasks on ImageNet, Imagen, COCO-S, and Dreambooth datasets.\n\n|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|4.5%|10.0%|3.0%|4.5%|6.0%|**72.0%**|\n\nWe test the computational efficiency of SINE on NVIDIA Tesla V100, and the remaining methods on NVIDIA GeForce RTX3090. It can be seen that our method is comparable to other methods in both computational time and memory cost.\n\nTable r3-4: Computational time and memory cost.\n\n|Method|SDEdit|DiffEdit|SINE|DDS|InstructPix2Pix|EDICT|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time per iteration (s)$\\downarrow$|**3**|9|3480|46|12|648|6|\n|Memory (GB)$\\downarrow$|10.0|**6.7**|28.0|16.7|18.0|13.8|**6.7**|\n\n**Q5: Difference between the editing and image-to-image translation task and the way to incorporate \"cat\" and \"dog\" classes into the translation task.** \n\nImage editing is to modify source images under specific guidance while the image-to-image translation aims to learn the mapping between two visual domains.\n\nFor translation, the \"cat\" images are used in the forward process, and \"dog\" images are used to train the specific pre-trained diffusion model for generating dog images without using the text information.\n\n**Q6: The paper should’ve been written in a very different way by taking inspiration from [Lugmayr A, et al., CVPR2022].**\n\nWe will cite the paper and take inspiration from the paper in the final version preparation, if accepted.\n\n**Q7: Discussing limitation. Including failure case and their possible explanation in the main paper.** \n\nA limitation of our method is that incorrect weighting matrix may lead to the failure of the method. Moreover, our method relies on a pre-trained diffusion model. Artifacts are produced when the desired edit involves generation failure cases of the underlying model. We will include limitations and failure cases in the main paper."}, {"review_id": "sxrl5vTV7Z", "reviewer": "Reviewer_rj2K", "summary": "The proposed model use differentiated reverse sampling strategy for image editing and translation.", "questions": "See the weakness part.", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed model use differentiated reverse sampling strategy for image editing and translation.", "weaknesses": "1. Please include the user study results on evaluating the natural image editing for qualitative evaluation.\n\n2. The method lacks novelty. Using different starting point with masked and non-masked region is just a variation of DiffEdit approach. Please elaborate the difference between baseline models. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Please include the user study results on evaluating the natural image editing for qualitative evaluation.**\n\nWe conduct user study by inviting 40 participants and providing each of them with 30 randomly selected source images where the corresponding generated results of different methods are displayed randomly. Participants were asked to choose the image that better applies the requested edit while preserving most of the original image details. We newly added three SoTA methods DDS [Hertz A, et al., arXiv:2304.07090, 2023], InstructPix2Pix [Brooks T, et al., CVPR2023] and EDICT [Wallace B, et al., CVPR2023] for comparison. The percentage of votes for each method is shown in Tables r4-1 and r4-2. The results demonstrate that the participants exhibit a strong preference for our method. \n\nTable r4-1: User study results of Cat $\\rightarrow$ Dog task.\n\n|ILVR|SDEdit|EGSDE|Ours|\n|:-:|:-:|:-:|:-:|\n|11.5%|10.5%|12.5%|**65.5%**|\n\nTable r4-2: User study results of the tasks on ImageNet, Imagen, COCO-S, and Dreambooth datasets.\n\n|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|4.5%|10.0%|3.0%|4.5%|6.0%|**72.0%**|\n\n**Q2: The method lacks novelty. Using different starting point with masked and non-masked region is just a variation of DiffEdit approach. Please elaborate the difference between baseline models.**\n\nFirst, our method and the DiffEdit method have different motivations. DiffEdit automatically generates a mask to indicate the regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Then use the inferred mask to replace the background with pixel values from the encoding process at the corresponding timestep. Differently, we add noise to each pixel based on its relevance to the editing task to achieve controllable editing. We establish the relationship between the noise variance and the timestep through rigorous theoretical deducing, and propose to denoise at different timesteps for different pixels to utilize the pre-trained isotropic Gaussian diffusion model. \n\nSecond, in terms of algorithm implementation, we are in a soft way compared with DiffEdit which uses the hard mask to guide the denoising process. Our algorithm starts the denoising process for a few pixels, and then gradually more pixels are included in the denoising process. Each pixel takes the denoising timestep according to its relevance to the editing task. This helps to generate more natural images and avoids artifacts caused by hard masks. \n\nFinally, in terms of generation quality, our method can generate images with more natural appearance. DiffEdit may generate unsmooth images around the boundary of the mask. For example, the images generated by DiffEdit shown in the first and fourth columns of Figure 3 in the main text have unnatural combination of the background and foreground.\n\nWe will clarify more on these differences in revision."}, {"review_id": "V10lLHcHdf", "reviewer": "Reviewer_ev4D", "summary": "This paper presents a novel Non-isotropic Gaussian Diffusion Model (NGDM) for image-to-image translation and image editing tasks. The central idea of NGDM is to add independent Gaussian noises with different variances to different pixels, thus achieving controllable image translation and editing based on the amount of noise variance added to each pixel. Unlike prior models, the NGDM doesn't need specific training; instead, it rectifies into an isotropic Gaussian diffusion model where different pixels have varying total forward diffusion time.\n\nTo generate images from the diffused ones, the authors propose a sampling method that initiates denoising at different times for different pixels using a pre-trained isotropic Gaussian diffusion model. This process allows for the preservation of certain parts of the image while editing or translating others, thereby providing flexibility in performing such tasks.\n\nThe paper demonstrates the effectiveness of NGDM through experiments on three datasets encompassing real and synthetic images. The results show that NGDM outperforms state-of-the-art score-based diffusion models (SBDMs) in terms of FID and SSIM metrics for Cat → Dog image translation task. Similar superior performance was also observed in text-guided image translation and image editing tasks.\n\nIn summary, the main contributions of the paper are:\n1. The proposal of a novel Non-isotropic Gaussian Diffusion Model (NGDM) for image-to-image translation and image editing.\n2. Demonstration of how to rectify the NGDM into an isotropic Gaussian diffusion model, which allows leveraging a pre-trained isotropic model for denoising and image generation.\n3. A new sampling method for generating images by starting denoising at different times for different pixels.\n4. Validation of the proposed model against state-of-the-art SBDMs across several tasks and datasets, showing improved performance.\n5. Exploration of the trade-off between image fidelity and alignment with desired translation/editing targets, providing an avenue for controlling this balance with varying hyperparameters.", "questions": "Here are a few questions and suggestions that would benefit from further clarification by the authors:\n\n1. **Noise Variance Distribution:** How is the variance of the Gaussian noise chosen for each pixel? Is there any strategy or algorithmic process behind this, or is it random? Knowing more about this could help understand how the NGDM manages to preserve certain parts of an image while translating/editing others.\n\n2. **Comparison with Other Models:** Could you provide a more detailed comparison between NGDM and other state-of-the-art models? It would be beneficial to include both qualitative (visual comparisons) and quantitative (additional metrics) results where possible.\n\n3. **Performance on Different Scenarios:** How does the model perform when applied to different tasks, especially those that have not been covered in the experimental section? Understanding its versatility and limitations across various scenarios will paint a fuller picture of NGDM's applicability.\n\n4. **Computational Efficiency:** Could you elaborate on the computational efficiency of your model? Specifically, how does the time complexity of the proposed method compare to other isotropic diffusion models? Does adding noise with different variances to different pixels significantly increase computational cost?\n\n5. **Ethical Considerations:** As your model allows substantial manipulation of images, what ethical considerations should be taken into account? Discussions around potential misuse, data privacy, and consent would add value.\n\n6. **Potential Improvements:** Lastly, can you suggest potential avenues for further improving the performance of NGDM? This might include parameter tuning, integrating with other models, or even extending the model to other domains beyond image translation and editing.\n\nLooking forward to your responses to these queries and suggestions, which I believe will provide a clearer understanding of your work and its implications.", "limitations": "Based on the provided information, it appears that the authors have not thoroughly addressed the potential limitations and negative societal impacts of their work.\n\nHere are a few suggestions for improvement:\n\n**Limitations:**\n\n1. **Computational Efficiency:** The authors should address the computational efficiency of their proposed model. Adding noise with different variances to different pixels might increase computational cost, which could be a limitation in some scenarios.\n   \n2. **Generalizability:** The paper does not discuss the performance of the model when applied to other tasks or datasets beyond those tested. Addressing this would give a clearer picture of the model's versatility and robustness.\n   \n3. **Model Interpretability:** The interpretability of the NGDM also seems unclear. Understanding why specific noise variances are assigned to certain pixels during the diffusion process could be complex. Discussing these aspects would help readers better understand and apply the model.\n\n**Societal Impacts:**\n\n1. **Ethical Considerations:** As the proposed model allows substantial manipulation of images, it is important to consider the ethical implications. The authors should provide a discussion about potential misuse, data privacy, and consent.\n\n2. **Potential Misuse:** With advanced image editing and translation capabilities, there may be potential for misuse of the technology, such as deepfake creation or unauthorized alteration of images. The authors should address these concerns and possible measures to prevent misuse.\n\nDiscussing these potential limitations and societal impacts would create a more balanced view of NGDM and help prepare users for any challenges they might face when applying the model.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality:** The proposed Non-isotropic Gaussian Diffusion Model (NGDM) demonstrates a high level of originality. It creatively deviates from the standard practice of isotropic diffusion by adding independent Gaussian noises with different variances to different pixels, enabling more controllable image translation and editing. This concept is novel and adds an interesting dimension to image generation models.\n\n**Quality:** The paper is of high quality. The authors provide a comprehensive explanation of the model, including detailed descriptions of the methodology. They describe the process of rectifying NGDM into an isotropic Gaussian diffusion model, which allows leveraging a pre-trained model for denoising and image generation. The experimental results on multiple datasets convincingly support their claims, showing that their method outperforms state-of-the-art score-based diffusion models in several tasks.\n\n**Clarity:** The paper is well-structured and clear. The authors do an excellent job explaining complex concepts in an understandable manner, making the paper accessible to readers with varying levels of familiarity with the topic. Diagrams and visual aids would further enhance the clarity of the paper.\n\n**Significance:** This work holds significant potential for advancing the field of image-to-image translation and image editing. The method provides a new way to control the translation/editing process by varying the noise added to different pixels, which could be invaluable in numerous applications. Moreover, this model's superior performance across multiple tasks and datasets shows promise for practical use in real-world scenarios. By demonstrating how to leverage existing isotropic Gaussian diffusion models within the NGDM framework, the authors also open up possibilities for future research in this area.", "weaknesses": "While the paper presents a novel and promising approach in the Non-isotropic Gaussian Diffusion Model (NGDM) for image-to-image translation and image editing tasks, there are several areas that could be improved:\n\n1. **Contextual Understanding:** The authors might consider providing more information on how their work compares to other existing methods. While they mention a few specific models and techniques, a more extensive literature review and comparison would be beneficial. This would allow readers to better understand the novelty and advantages of NGDM.\n\n2. **Experimentation:** The experiments conducted show promising results; however, the testing could be expanded. The authors only use three datasets for validation. More diverse datasets could help provide a more comprehensive performance evaluation of the method across different scenarios. Also, it would be helpful if the authors could include comparative visual results along with the quantitative metrics.\n\n3. **Model Interpretability:** While the model's performance is impressive, its interpretability seems not addressed. It might be challenging to understand why specific noise variances are assigned to certain pixels during the diffusion process. Providing some insights or analysis into this aspect may make the model more understandable.\n\n4. **Potential Limitations and Pitfalls:** The paper lacks a discussion about potential limitations and considerations when using the proposed model. Addressing possible issues such as computational cost, scalability, and any conditions under which the model might fail or perform suboptimally would provide a more balanced and realistic view of the proposed method. \n\n5. **Societal Impact:** As with any machine learning model, it's crucial to consider the ethical implications and potential misuse cases. Given the model's capacity for image manipulation, discussions around data privacy and consent, as well as potential ways the technology could be misused, should be included.\n   \nThese suggestions should help to strengthen the paper and broaden the understanding and applicability of the proposed NGDM.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Comparison with more methods and experiments on more datasets.** \n\nFor Cat $\\rightarrow$ Dog translation task, we add SDDM [Sun S, et al., ICML2023] for comparison. SDDM decomposes the score function into an image “denoising” part and a content “refinement” part for translation. Differently, we perform translation by adding independent noise with different variances to different pixels. Results in Table r5-1 show that our method achieves the best results among the compared methods.\n\nTable r5-1: Quantitative comparison on Cat $\\rightarrow$ Dog translation task. \n\n|Method|ILVR|SDEdit|EGSDE|SDDM|NGDM (Ours)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|FID($\\downarrow$)|74.37±1.55|74.17±1.01|65.82±0.77|62.29±0.63|**61.39±0.27**|\n|SSIM($\\uparrow$)|0.363±0.001|0.423±0.001|0.415±0.001|0.422±0.001|**0.478±0.001**|\n\nFor image editing task, we add DDS [Hertz A, et al., arXiv:2304.07090, 2023], InstructPix2Pix [Brooks T, et al., CVPR2023] and EDICT [Wallace B, et al., CVPR2023] for comparison. DDS utilizes delta scoring to provide effective gradients for editing. InstructPix2Pix trains a conditional diffusion model for editing that enables zero-shot generalization. EDICT proposes exact inversion of real and generated images. Differently, we edit the image by adding independent noise with different variances to different pixels. \n\nWe additionally consider two natural datasets (COCO-S and DreamBooth Dataset [Ruiz N, et al., CVPR2023]). The qualitative results are shown in Figure r1 in the uploaded one-page pdf. The quantitative results shown in Table r5-2 indicate that our method outperforms competing methods by achieving a better trade-off between CLIPScore and LPIPS value.\n\nTable r5-2: Quantitative comparison with different methods on four datasets.\n\n|Method|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours ($a$=10.0,$b$=5.0)|Ours ($a$=10.0,$b$=6.0)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CLIPScore$\\uparrow$ (Imagenet)|30.39|28.98|28.13|27.64|29.28|**30.66**|29.61|\n|LPIPS$\\downarrow$ (Imagenet)|58.73|32.40|43.57|44.93|41.41|37.10|**31.32**|\n|CLIPScore$\\uparrow$ (Imagenen)|35.81|35.19|34.69|34.75|**36.06**|35.66|35.53|\n|LPIPS$\\downarrow$ (Imagenen)|50.84|21.81|31.38|**19.31**|46.07|24.29|20.28|\n|CLIPScore$\\uparrow$ (COCO-S)|28.58|30.31|30.48|30.77|31.28|**31.75**|31.45|\n|LPIPS$\\downarrow$ (COCO-S)|55.76|29.60|31.53|24.00|36.55|28.80|**23.43**|\n|CLIPScore$\\uparrow$ (Dreambooth)|19.64|19.88|**20.17**|19.70|19.64|19.81|19.75|\n|LPIPS$\\downarrow$ (Dreambooth)|59.82|27.47|33.26|22.52|40.50|24.86|**19.95**|\n\n**Q2: Model interpretability.**\n\nImage editing task aims to modify the source image under target guidance while leaving the regions that are unrelated to the editing task unchanged.\n\nIt is empirically known that the diffusion model can generate more diverse novel content if adding noise with larger variance to the image, and preserve the content if adding smaller variance noise. Motivated by this, we achieve controllable editing by adding different variances of noise to different pixels. \n\nAs illustrated in the lower part of Figure 1, with $\\mathbf{\\Lambda}(\\mathcal{I})$ computed on the source image by method DiffEdit in Section 3.4, the nose region is given with large variance noise for translating cat to dog, while the background is preserved by adding small variance noise. \n\n**Q3: Potential limitations and improvements.** \n\nA limitation of our method could be that incorrect weighting matrix may lead to the failure of the method. Moreover, our method relies on a pre-trained diffusion model. Artifacts are produced when the edit involves generation failure cases of the underlying model. We will add the limitations in the paper.\n\nIn future work, we will design a better way to calculate the weighting matrix more precisely and efficiently.\n\n**Q4: Societal impact, ethical considerations and potential misuse.** \n\nIn our experiments, all the considered datasets are open-sourced and publicly available. Our work aims to manipulate images with minimum effort. However, this method might be misused by faking images. We will take care to exploit the method to avoid the potential negative social impact and we will help research in identifying and preventing malicious editing. To mitigate potential misuse, we will release our code under a license focused on ethical and legal use, stating explicitly that illegal and unethical use will not be allowed.\n\n**Q5: Noise Variance Distribution.** \n\nWe described in Section 3.4 of the main text about how to choose the variance of the Gaussian noise for each pixel.\n\n**Q6: Performance on different scenarios.**\n\nWe perform two tasks that were not covered in the experimental section, including local style transfer and gender transformation. The local style transfer task aims to transform the specified object in the image into another style while preserving the content of the rest of the region. Gender transformation aims to transform male into female. Due to space limitations, we show a few examples in Figure r2 of the uploaded one-page pdf, and we will show more examples in Appendix B in the revised version.\n\n**Q7: Computational efficiency.** \n\nTables r5-4 and r5-5 show the computational time and memory cost of different methods. Our method is comparable to the other methods in both computational time and memory cost.\n\nTable r5-4: Computation time and memory of methods with image space-based diffusion model.\n\n|Method|SDEdit|ILVR|EGSDE|DDIB|DiffuseIT|NGDM (Ours)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time per iteration (s)$\\downarrow$|**18**|44|62|210|48|42|\n|Memory(GB)$\\downarrow$|3.3|**2.8**|4.5|3.8|16.6|7.4|\n\nTable r5-5: Computation time and memory of methods with latent space-based diffusion model.\n\n|Method|SDEdit|DiffEdit|SINE|DDS|InstructPix2Pix|EDICT|NGDM (Ours)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time per iteration (s)$\\downarrow$|**3**|9|3480|46|12|648|6|\n|Memory(GB)$\\downarrow$|10.0|**6.7**|28.0|16.7|18.0|13.8|**6.7**|"}, {"review_id": "oXWHMELGPM", "reviewer": "Reviewer_LtuA", "summary": "Gaussian diffusion is a common concept used in image processing, physics, and machine learning. It refers to a process where values (like the pixels in an image) are \"smoothed out\" or diffused according to a Gaussian function (also known as a normal distribution or bell curve), which provides a natural and mathematically convenient model for this diffusion process.  In the context of Gaussian diffusion, the term \"isotropic\" means that the diffusion is uniform in all directions. In contrast, \"non-isotropic\" Gaussian diffusion means that the diffusion is not uniform in all directions. The amount a pixel gets diffused might depend on its direction. \n\nThe authors propose a method to use a pre-trained Isotropic Gaussian Diffusion Model (IGDM) for sampling in the context of a Non-isotropic Gaussian Diffusion Model (NGDM). In the first step, they define the NGDM with added independent non-isotropic Gaussian noise. This suggests that they are introducing a type of Gaussian noise that's directionally dependent, in line with the concept of non-isotropic diffusion. Then, they detail how to implement the NGDM using the pre-trained IGDM. This process involves 'rectifying' the spatially different times of the noise and denoise procedures in the NGDM. It seems they're addressing the inherent differences between isotropic and non-isotropic models and adjusting the processes accordingly. Finally, they present a data sampling algorithm for the proposed NGDM that uses the pre-trained IGDM. This suggests that they're leveraging the pre-trained model's capabilities in the new context of non-isotropic diffusion.\n\n", "questions": "1. Please compare computational cost of all involved methods resented in Section 4.\n2. Please show a couple of examples where NGDM performs worse, and give the reasons to explain why.", "limitations": "It is simply impossible to thoroughly \"prove\" a method is better by just displaying some selected results.  A much more robust and comprehensive empirical study followed by why the proposed NGDM works better would be helpful.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Overall, they're taking an existing model trained on isotropic diffusion, adapting it for non-isotropic diffusion, and presenting a new algorithm for sampling data within this context.\n\n2. The authors show through experiments that their proposed method to be competitive.  ", "weaknesses": "1. The authors discussed the settings of the parameters \\alpha and \\beta but I do not see a strong link to the quality of the generated images.\n\n2. From the generated images shown in Figures 2-4, 6, I cannot really say that the proposed method outperforms the other consistently.  There is simply no \"wow\" effect.  Figure 6 shows that the color of the head also changed, not described in \"black leather jacket\".\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The link of $a$ and $b$ to the quality of the generated images.** \n\nWe present the visual results when the hyper-parameters $a$ and $b$ vary respectively in Figure 6 of the main paper. Keeping $b$ constant, when $a$ is small, all element values of the weighting matrix obtained after Sigmoid transformation are relatively small. At this point, the noise variance on each pixel of the image is small, resulting in that the generated image is similar to the source image. However, the generated image does not align well with the target prompt. When $a$ gets larger, the generated image begins to align with the target prompt. Keeping $a$ constant, $b$ does the opposite. When $b$ is small, the generated image does not preserve well the source image information. When $b$ gets larger, the generated image will preserve more source image information.\n\n**Q2: The proposed method does not outperform the other consistently. Figure 6 shows that the color of the head also changed, not described in \"black leather jacket\".**\n\nWe focus on controllable image editing that edit image based on target prompt while minimally modifying the source image. Compared with other methods, our method can better preserve the background, pose, etc. For example, for images with complex backgrounds in columns 2-5 of Figure 2, our method can accurately keep the background of the source image unchanged while translating cats into dogs. Other methods either blur the background or fail to maintain the source image background. In Figure 3, the source images in columns 2 and 4 have detailed backgrounds, and our method can preserve these details to the greatest extent while editing images.\n\nAs we analyzed above and the results shown in Figure 6 of the main paper, when $a$ is 20 or $b$ is 0, the source image information cannot be well maintained, resulting in changes in the head's color. We can achieve a balance between the two by choosing appropriate parameters. When $a=10.0$ and $b=5.0$, our method can preserve the head color described in \"black leather jacket\".\n\n**Q3: Computational cost comparison.**\n\nTables r6-1 and r6-2 show the computational efficiency comparison. Our method has relatively small inference time and requires no additional training.\n\nTable r6-1: Computational time and memory cost of methods with image space-based diffusion model.\n\n|Method|SDEdit|ILVR|EGSDE|DDIB|DiffuseIT|NGDM (Ours)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time per iteration (s)$\\downarrow$|**18**|44|62|210|48|42|\n|Memory(GB)$\\downarrow$|3.3|**2.8**|4.5|3.8|16.6|7.4|\n\nTable r6-2: Computational time and memory cost of methods with latent space-based diffusion model.\n\n|Method|SDEdit|DiffEdit|SINE|DDS|InstructPix2Pix|EDICT|NGDM (Ours)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time per iteration (s)$\\downarrow$|**3**|9|3480|46|12|648|6|\n|Memory(GB)$\\downarrow$|10.0|**6.7**|28.0|16.7|18.0|13.8|**6.7**|\n\n**Q4: A couple of examples where NGDM performs worse and reasons for explaining why.**\n\nWe show a few failure examples in Figure 6 of the Appendix. We think that the failure of the image in the first column may be due to the artifacts generated by the underlying model we depend on, anf the failures of images in columns 2 and 3 may be due to that the computed weighting matrix is not accurate.\n\n**Q5: A much more robust and comprehensive empirical study followed by why the proposed NGDM works better.** \n\nWe achieve controllable image editing by adding a corresponding degree of noise to each pixel to the extent that it needs to be edited. Our method can effectively preserve the original content by adding noise with small variances to the regions irrelevant to the editing task. Compared with mask-guided image editing, such as DiffEdit, our method can avoid the edge artifact problem caused by the mask.\n\nWe add more experimental results to justify the effectiveness of our proposed NGDM, including the comparison results on more datasets and more SoTA methods, the results of user study, and the results of computation efficiency. \n\nWe add two natural datasets (COCO-S and DreamBooth Dataset [Ruiz N, et al., CVPR2023]), and three SoTA methods DDS [Hertz A, et al., ], InstructPix2Pix [Brooks T, et al., CVPR2023] and EDICT [Wallace B, et al., CVPR2023] for comparison. The quantitative results shown in Table r6-3 indicate that our method outperforms competing methods by achieving a better trade-off between CLIPScore and LPIPS value. \n\nTable r6-3: Quantitative comparison on COCO-S dataset and Dreambooth dataset.\n\n|Method|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours ($a$=10.0,$b$=5.0)|Ours ($a$=10.0,$b$=6.0)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CLIPScore$\\uparrow$ (COCO-S dataset)|28.58|30.31|30.48|30.77|31.28|**31.75**|31.45|\n|LPIPS$\\downarrow$ (COCO-S dataset)|55.76|29.60|31.53|24.00|36.55|28.80|**23.43**|\n|CLIPScore$\\uparrow$ (Dreambooth dataset)|19.64|19.88|**20.17**|19.70|19.64|19.81|19.75|\n|LPIPS$\\downarrow$ (Dreambooth dataset)|59.82|27.47|33.26|22.52|40.50|24.86|**19.95**|\n\nWe conduct user study by inviting 40 participants and providing each of them with 30 randomly selected source images where the corresponding generated results of different methods are displayed randomly. Participants were asked to choose the image that better applies the requested edit while preserving most of the original image details. The percentage of votes for each method is shown in Tables r6-4 and r6-5, which demonstrate that the participants exhibit a strong preference for our method. \n\nTable r6-4: User study results on Cat $\\rightarrow$ Dog task.\n\n|ILVR|SDEdit|EGSDE|Ours|\n|:-:|:-:|:-:|:-:|\n|11.5%|10.5%|12.5%|**65.5%**|\n\nTable r6-5: User study results on the remaining tasks.\n\n|SDEdit|DiffEdit|DDS|EDICT|InstructPix2Pix|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|4.5%|10.0%|3.0%|4.5%|6.0%|**72.0%**|"}]

Author rebuttal:
Dear ACs and reviewers,

Thanks for the insightful comments and suggestions on our paper. We have carefully responded to the comments of each reviewer. Meanwhile, we have uploaded a PDF file to show visual results as support material. We will revise our paper accordingly in the final version if accepted.

Best,

Authors

Meta-review: The paper presents a method for image-to-image translation and image editing called Non-isotropic Gaussian Diffusion Model (NGDM). NGDM is a score-based diffusion model that can be used to generate images that are both faithful to the source image and aligned with the desired translation/editing target. NGDM is constructed by adding independent Gaussian noises with different variances to different image pixels. The paper shows that NGDM can be used to achieve impressive results on both image-to-image translation and image editing tasks. 

While the reviewers are not in uniform agreement that the paper should be published, I believe the authors have done an adequate and thorough job in addressing all the concerns, to the extent where no major concerns about the overall merit of the work should remain. Overall, I judge the paper to be a sufficiently strong contribution to be published.
Decision: Accept (poster)