SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xi  $\mathbf{Y}\mathbf{u}^{*}$  , Xiang Gu\*, Haozhi Liu, Jian Sun (
School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an, China {ericayu, xianggu, liuhzh}@stu.xjtu.edu.cn {jiangsun}@xjtu.edu.cn
Score-based diffusion models (SBDMs) have achieved state-of-the-art results in image generation. In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing, which requires editing the source image while preserving the image regions irrelevant to the editing task. We construct NGDM by adding independent Gaussian noises with different variances to different image pixels. Instead of specifically training the NGDM, we rectify the NGDM into an isotropic Gaussian diffusion model with different pixels having different total forward diffusion time. We propose to reverse the diffusion by designing a sampling method that starts at different time for different pixels for denoising to generate images using the pre-trained isotropic Gaussian diffusion model. Experimental results show that NGDM achieves state-of-the-art performance for image editing tasks, considering the trade-off between the fidelity to the source image and alignment with the desired editing target.
Score-based diffusion models (SBDMs) [1-6] demonstrate state-of-the-art performance on image synthesis quality and sample diversity. SBDMs are widely applied to applications such as text-to-image synthesis [7-9], image editing [10-15], deblurring [16, 17], etc. SBDMs consist of a forward diffusion stage that adds random noise to data and a reverse stage that generates desired data from noise. The introduced noise in the forward process is commonly isotropic Gaussian noise [1, 6], i.e., independently and identically distributed noise in a normal distribution.
Non-isotropic diffusion model, by adding non-isotropic noises in the forward diffusion process has been investigated in [18-22]. The blurring diffusion model in [18] adds blur and noise to samples which is a Gaussian diffusion process with non-isotropic noise in the frequency space. [19] employs auxiliary velocity variables to augment the data variables as Hamiltonian dynamics, and performs the diffusion process in this expanded joint space by adding different noises to the auxiliary and the data variables. [20] formulates the diffusion model using non-isotropic noise with a positive semi-definite covariance matrix, and carries out a comparative analysis of the non-isotropic and isotropic diffusion models. These works have shown better performance for data generation.
This paper focuses on image editing tasks that commonly require editing specific object/thing of an image while preserving the remaining parts of the image. For image editing, [11] produces a mask that allows the preservation of context while editing the remaining part. [23-25] use the learned text embedding for the object that needs to be preserved in the image to ensure the object is unchanged during editing. It is empirically known that the diffusion model can generate more diverse novel
content if adding noise with larger variance to the image while preserving the image content if adding smaller variance noise [13]. Motivated by this, we employ a non-isotropic diffusion model to add noises with different variances to different image pixels, considering the degree to which the corresponding pixels should be edited/preserved.
Along with this idea, we present a Non-isotropic Gaussian Diffusion Model (NGDM), utilizing an off-the-shelf isotropic diffusion model (e.g., DDPM [1]) for achieving the data sampling in image editing. Specifically, given a source image, the proposed NGDM is with a diffusion process that adds independent Gaussian noises with different variances to different pixels, therefore the added noise is independent and non-isotropic over image pixels. We then employ an off-the-shelf isotropic diffusion model to execute the reverse denoising process for image editing. To achieve this goal, we rectify the NGDM into the isotropic Gaussian diffusion model where each pixel is added with the same amount of noise at each step but different pixels have varying total number of noise accumulation time steps. We subsequently devise a specific sampling method for NGDM that can generate images by using the pre-trained isotopic Gaussian diffusion model.
We demonstrate the effectiveness of NGDM in image editing tasks on five datasets including real and synthetic datasets. In the experiments for cats to dogs editing task on AFHQ dataset, our method achieves the best performance in the metric of FID and SSIM compared with the SoTA SBDMs-based methods. For text-guided image editing, our method achieves better trade-off between CLIPScore and LPIPS value. Furthermore, NGDM allows for flexible trade-off with varying hyper-parameters.
2 Background: Score-based Diffusion Models
SBDMs [1, 2, 4-6] are a family of generative models that learn the data distribution based on the Gaussian process. Two representative models are Denoising Diffusion Probabilistic Model (DDPM) [1] and Score Matching with Langevin Dynamics (SMLD) [5]. We discuss the details based on DDPM for the remainder of the main text for brevity.
Given the input data  $\mathbf{x}(0) \in \mathbb{R}^D$ , which represents a sample from the data distribution  $p_{data}$ , a forward process produces the noisy  $\mathbf{x}(t)$  indexed by a time variable  $t \in [0,1]$  via
$$
\mathbf {x} (t) = \sqrt {\bar {\alpha} (t)} \mathbf {x} (0) + \sqrt {1 - \bar {\alpha} (t)} \mathbf {z} (t), \tag {1}
$$
where  $\mathbf{z}(t) \in \mathcal{N}(\mathbf{0},\mathbf{I})$  for any  $t$  and  $\bar{\alpha}(t) = e^{-\int_0^t \beta(s)\mathrm{d}s}$  controlling the noise schedule.  $\beta(s) = \bar{\beta}_{\min} + s(\bar{\beta}_{\max} - \bar{\beta}_{\min})$  with  $\bar{\beta}_{\min} = 0.1$  and  $\bar{\beta}_{\max} = 20$  [1, 6]. This type of diffusion model is dubbed Isotropic Gaussian Diffusion Model (IGDM), since the added Gaussian noise  $\mathbf{z}(t)$  is from the independently and identically distributed normal distribution.
DDPM is in the framework of Stochastic Differential Equations (SDEs) [5] with variance preservation
$$
\mathrm {d} \mathbf {x} (t) = - \frac {1}{2} \beta (t) \mathbf {x} (t) \mathrm {d} t + \sqrt {\beta (t)} \mathrm {d} \mathbf {w} \quad \text {w i t h i n i t i a l v a l u e} \mathbf {x} (0), \tag {2}
$$
where  $\mathbf{w}$  is the standard Wiener process. The reverse process denoises the noisy sample  $\mathbf{x}(T)$  starting from  $T$  using a reverse SDE
$$
\mathrm {d} \mathbf {x} (t) = \left[ - \frac {1}{2} \beta (t) \mathbf {x} (t) - \beta (t) \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x} (t)) \right] \mathrm {d} t + \sqrt {\beta (t)} \mathrm {d} \bar {\mathbf {w}} \quad \text {w i t h i n i t i a l v a l u e} \mathbf {x} (T), \tag {3}
$$
where  $\bar{\mathbf{w}}$  is a standard Wiener process when time flows backward from  $T$  to 0. The score function  $\nabla_{\mathbf{x}}\log p_t(\mathbf{x})$  is approximated by training a time-dependent model  $\mathbf{s}_{\theta}(\mathbf{x}(t),t)$  via score matching [6, 26]. For inference, the time of the differential equation is discretized as  $t\in \{0,\Delta t,2\Delta t,\dots ,T\}$  with  $\Delta t$  representing the sampling time interval. We can choose to utilize the reverse process of DDPM or the reverse process of DDIM for sampling. With  $\beta_{t} = \beta (t)\Delta t$ , the iteration rule of DDPM [1] is
$$
\mathbf {x} (t) = \frac {1}{\sqrt {1 - \beta_ {t + \Delta t}}} \left(\mathbf {x} (t + \Delta t) + \beta_ {t + \Delta t} \mathbf {s} _ {\boldsymbol {\theta}} (\mathbf {x} (t + \Delta t), t + \Delta t)\right) + \sqrt {\beta_ {t + \Delta t}} \mathbf {z} (t + \Delta t), \tag {4}
$$
where  $\mathbf{z}(t + \Delta t) \in \mathcal{N}(\mathbf{0},\mathbf{I})$ . With  $\bar{\alpha}_t = \prod_{s=0}^t (1 - \beta_s)$ , the iteration rule of DDIM [27] is
$$
\begin{array}{l} \mathbf {x} (t) = \sqrt {\bar {\alpha} _ {t}} \left(\frac {\mathbf {x} (t + \Delta t) + \left(1 - \bar {\alpha} _ {t + \Delta t}\right) \cdot \mathbf {s} _ {\theta} (\mathbf {x} (t + \Delta t) , t + \Delta t)}{\sqrt {\bar {\alpha} _ {t + \Delta t}}}\right) \tag {5} \\ - \sqrt {(1 - \bar {\alpha} _ {t} - \sigma^ {2} (t + \Delta t)) (1 - \bar {\alpha} _ {t + \Delta t})} \cdot \mathbf {s} _ {\boldsymbol {\theta}} (\mathbf {x} (t + \Delta t), t + \Delta t) + \sigma (t + \Delta t) \mathbf {z} (t + \Delta t). \\ \end{array}
$$
Figure 1: The overview of our NGDM. We rectify the non-isotropic diffusion model into isotropic model with different total time steps (e.g.,  $T_{1},\dots ,T_{D}$ ) for different pixels. Based on this rectification, the input data  $\mathbf{y}(0)$  is firstly added isotropic noises until  $T$  time steps. Then in the reverse denoising process, to apply the pre-trained isotropic Gaussian diffusion model, we construct the noisy image to be denoised in each time step following Eq. (12). The red arrow in the figure indicates the pixel replacement operation in Eq. (12) when  $T_{k}\leq t\leq T$  for the  $k$ -th pixel at a denoising time step  $t$ .
In this section, we present a framework for utilizing the pre-trained Isotropic Gaussian Diffusion Model (IGDM) to achieve the sampling process of Non-isotropic Gaussian Diffusion Model (NGDM). In the following, first, we define the NGDM with added independent non-isotropic Gaussian noise. Then, we implement NGDM by IGDM through rectifying the spatially different time of noising and denoising procedures in NGDM and present our proposed data sampling algorithm for the proposed NGDM using the pre-trained IGDM.
3.1 Non-isotropic Gaussian Diffusion Model
In this work, we focus on the Non-isotropic Gaussian Diffusion Model (NGDM) by adding the nonisotropic Gaussian noise in the input data  $\mathbf{y}(0)\in \mathbb{R}^{D}$  and  $\mathbf{y}(0)\sim p_{data}$ , and the noises associated with different pixels are independent. The forward SDE of NGDM is
$$
\mathrm {d} \mathbf {y} = - \frac {1}{2} \beta (t) \boldsymbol {\Lambda} (\mathcal {I}) \mathbf {y} \mathrm {d} t + \sqrt {\beta (t) \boldsymbol {\Lambda} (\mathcal {I})} \mathrm {d} \mathbf {w} \quad \text {w i t h i n i t i a l v a l u e} \mathbf {y} (0), \tag {6}
$$
where  $\mathcal{I} \in \mathbb{R}^D$  is the source data,  $\Lambda(\mathcal{I}) : \mathbb{R}^D \to \mathbb{R}^{D \times D}$  is the weighting coefficient matrix, defined as diagonal matrix  $\Lambda(\mathcal{I}) = \operatorname{diag}(\lambda_1, \dots, \lambda_D)$  with  $0 \leq \lambda_k \leq 1$  scaling the Gaussian noise level added to the  $k$ -th pixel. Note that the transition kernel  $p_{0t}(\mathbf{y}(t) | \mathbf{y}(0)) = \mathcal{N}\left(\mathbf{y}(t) | \mathbf{y}(0) e^{-\frac{1}{2} \int_0^t \beta(s) \Lambda(\mathcal{I}) \mathrm{d}s}, \mathbf{I} - e^{-\int_0^t \beta(s) \Lambda(\mathcal{I}) \mathrm{d}s}\right)$  is an independent Gaussian distribution.
3.2 Rectify the Non-isotropic Gaussian Diffusion Model
With the added independent noise, we next discuss the forward SDE for NGDM in scalar form for each pixel  $k$ . Given the initial  $\mathbf{y}^k(0)$  denoting the value of pixel  $k$  in  $\mathbf{y}(0)$ , the forward SDE of the  $k$ -th pixel can be presented by
$$
\mathrm {d} \mathbf {y} ^ {k} = - \frac {1}{2} \beta (t) \lambda_ {k} \mathbf {y} ^ {k} \mathrm {d} t + \sqrt {\beta (t) \lambda_ {k}} \mathrm {d w} \quad \text {w i t h i n i t i a l v a l u e} \mathbf {y} ^ {k} (0), \tag {7}
$$
where  $\mathrm{w}$  is a one-dimensional Wiener process.
We present Theorem 1 to illustrate the connection between the NGDM defined in Section 3.1 and the IGDM defined in Section 2 at the pixel level. Beforehand, we introduce the following lemma.
Lemma 1. Let  $\beta(s) = \bar{\beta}_{\mathrm{min}} + s(\bar{\beta}_{\mathrm{max}} - \bar{\beta}_{\mathrm{min}})$  with  $\bar{\beta}_{\mathrm{max}} > \bar{\beta}_{\mathrm{min}} > 0$ . Then, for each  $\lambda_k \in [0,1]$  and  $t \in [0,1]$ , there exists a unique time  $\tau \in [0,1]$  (denoted by  $\tau = \xi_k(t)$ ) such that  $\int_0^t \beta(s) \lambda_k \, \mathrm{d}s = \int_0^\tau \beta(s) \, \mathrm{d}s$  and  $\beta(t) \lambda_k \, \mathrm{d}t = \beta(\tau) \, \mathrm{d}\tau$ , with the following form
$$
\xi_ {k} (t) = \frac {- \bar {\beta} _ {\operatorname* {m i n}} + \sqrt {\bar {\beta} _ {\operatorname* {m i n}} ^ {2} + 2 (\bar {\beta} _ {\operatorname* {m a x}} - \bar {\beta} _ {\operatorname* {m i n}}) \bar {\beta} _ {\operatorname* {m i n}} t \lambda_ {k} + (\bar {\beta} _ {\operatorname* {m a x}} - \bar {\beta} _ {\operatorname* {m i n}}) ^ {2} t ^ {2} \lambda_ {k}}{\bar {\beta} _ {\operatorname* {m a x}} - \bar {\beta} _ {\operatorname* {m i n}}}. \tag {8}
$$
The proof is in Appendix A. Based on the above Lemma, we can rectify the NGDM, which adds noise at each pixel with varying variance over the same time span, into an IGDM that adds noise at each pixel with the same noise variance but with different total diffusion time for different pixels. We introduce the following theorem to derive the differential equation as an IGDM.
Theorem 1. For a pixel indexed by  $k$ ,  $\lambda_{k} \in [0,1]$ , and let  $\tau = \xi_{k}(t)$  with  $\xi_{k}(t)$  represented in Eq. (8). With the same initial value  $\mathbf{y}^{k}(0)$ , we have that the transition kernel at time  $t$  induced by Eq. (7) is equal to the transition kernel at time  $\tau$  induced by the following differential equation
$$
\mathrm {d} \mathbf {y} ^ {k} = - \frac {1}{2} \beta (\tau) \mathbf {y} ^ {k} \mathrm {d} \tau + \sqrt {\beta (\tau)} \mathrm {d w} \quad \text {w i t h i n i t i a l v a l u e} \mathbf {y} ^ {k} (0). \tag {9}
$$
The total time of noising for Eq. (9) is  $T_{k}$  with  $T_{k} = \xi_{k}(T)$ .
We provide the proof in Appendix A. Inspired by this, we rectify the reverse process in NGDM with different speeds of denoising across pixels to be the reverse process with consistent speed but different total time of denoising. We suggest rectifying the differential equation for the reverse process of pixel  $k$  within the NGDM framework into the following form
$$
\mathrm {d} \mathbf {y} ^ {k} = \left[ - \frac {1}{2} \beta (\tau) \mathbf {y} ^ {k} - \beta (\tau) \left(\nabla_ {\mathbf {y}} \log p _ {\tau} (\mathbf {y})\right) ^ {k} \right] \mathrm {d} \tau + \sqrt {\beta (\tau)} \mathrm {d} \bar {\mathrm {w}} \quad \text {w i t h i n i t i a l v a l u e} \mathbf {y} ^ {k} \left(T _ {k}\right), \tag {10}
$$
where  $\bar{\mathbf{w}}$  is a one-dimensional Wiener process when time flows backward from  $T_{k}$  to 0. Theorem 1 establishes the conclusion that the NGDM in Eq. (7) can be rectified to the IGDM in Eq. (9) but with different total diffusion time  $T_{k}$  for different pixel indexed by  $k$ , determined based on Eq. (8). This inspires us to utilize the pre-trained IGDM to achieve the data sampling of NGDM for image editing. Subsequently, we present a method that adjusts the total time of noising and denoising for each pixel  $k$  to  $T_{k}$ , enabling us to use the pre-trained IGDM for data sampling. The corresponding sampling method is presented in Algorithm 1.
For image editing tasks, we use the source image  $\mathcal{I}$  as  $\mathbf{y}(0)$  and generate noisy data  $\mathbf{y}(T)$  through the forward process. We generate the edited image  $\hat{\mathbf{y}}(0)$  by denoising from  $\mathbf{y}(T)$ . Utilizing the forward noiseing process of IGDM, we add independent noise to each pixel  $k$  to obtain the noisy observation  $\mathbf{x}^k(t)$  of discrete time  $t \in \{0, \Delta t, \dots, T\}$  with  $\Delta t$  representing the sampling time interval
$$
\mathbf {x} ^ {k} (t) = \sqrt {\bar {\alpha} _ {t}} \mathcal {I} ^ {k} + \sqrt {1 - \bar {\alpha} _ {t}} \mathbf {z} ^ {k} (t), \tag {11}
$$
where  $\mathbf{z}(t) \in \mathcal{N}(\mathbf{0},\mathbf{I})$ . Next, with  $\mathbf{H}(\mathbf{y}(t + \Delta t),t + \Delta t)$  denotes the sampling procedure of DDPM and DDIM given in Eq. (4) and Eq. (5) of Section 2, the data sampling iteration utilizing the IGDM model with initial value  $\mathbf{y}^k (T) = \mathbf{x}^k (T)$  is defined as
$$
\mathbf {y} ^ {k} (t) = \left\{ \begin{array}{l l} \mathbf {x} ^ {k} (t) & T _ {k} \leq t <   T \\ \mathbf {H} ^ {k} (\mathbf {y} (t + \Delta t), t + \Delta t) & \text {o t h e r w i s e .} \end{array} \right. \tag {12}
$$
This implies that we use the noisy observation  $\mathbf{x}^k (t)$  to represent  $\mathbf{y}^k (t)$  at each step before  $T_{k}$  with  $t\geq T_k$  , rather than the actual denoised result starting from time step  $T$  . Until time  $T_{k}$  we begin the denoising from  $\mathbf{x}^k (T_k)$  for  $k$  -th pixel. In such a way, different pixels have different starting time steps ( $T_{k}$  for  $k$  -th pixel) for image denoising in the data sampling process.
3.3 Sampling Method in NGDM
Based on the above method, we further specify our sampling algorithm by utilizing a pre-trained IGDM. We do not require the training of NGDM, but instead harness the power of a pre-trained IGDM for data sampling. We generate the edited image with the source image  $\mathcal{I}$  as a condition. We first add noise to the source image  $\mathcal{I}$  to  $T$  time steps as the starting point of denoising, and then use the method in Section 3.2 to rectify NGDM into IGDM to denoise the image using the pre-trained IGDM. We show the sampling algorithm of NGDM in the Algorithm 1.
Algorithm 1 Sampling Method of NGDM
Inputs: The source image  $\mathcal{I}$ , the weighting matrix  $\Lambda (\mathcal{I})$ , the score function  $s_\theta$ , the time schedule  $\{\beta (t)\}_{t = 0}^T$ , the maximal time steps  $T$   
1: Compute  $\bar{\mathbf{T}} = [\xi_1(T),\dots ,\xi_D(T)]$  according to Eq. (8)  
2:  $\mathbf{z}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$   
3:  $\mathbf{y} = \sqrt{\bar{\alpha}_T}\mathcal{I} + \sqrt{1 - \bar{\alpha}_T}\mathbf{z}$  # The starting point of denoising  
4: for  $t = T$  to 0 do  
5:  $\mathbf{x} = \sqrt{\bar{\alpha}_t}\mathcal{I} + \sqrt{1 - \bar{\alpha}_t}\mathbf{z}$  # Sample the noisy source image at time step  $t$   
6:  $\mathbf{M} = \mathbb{I}(t\geq \bar{\mathbf{T}})$   
7:  $\mathbf{y}\gets \mathbf{M}\odot \mathbf{x} + (\mathbf{1} - \mathbf{M})\odot \mathbf{y}$  # 1 is the  $D$ -dimensional all-ones vector  
8:  $\mathbf{y}\gets \mathbf{H}(\mathbf{y},t)$   
9: end for  
Output: Generated image  $\mathbf{y}$  conditioned on the source image  $\mathcal{I}$
3.4 Design of Input-dependent Weighting Matrix
We construct the weighting matrix  $\Lambda(\mathcal{I})$  based on the method for designing mask using a text-conditioned diffusion model [9] in DiffEdit [11]. Specifically, given the source image  $\mathcal{I}$ , the text description  $R$  of the source image, and the target description  $Q$  that describes the desired target image after editing. Following DiffEdit [11], we add noise to the source image up to  $0.5T$  step and use the texts  $R$  and  $Q$  as the conditions respectively for denoising in the current time step to estimate the score values by using the score network  $\mathbf{s}_{\theta}$ . We derive an attention map  $\mathcal{A}(\mathcal{I})$  based on the absolute difference of the estimated scores. We use the above method to compute 10 estimated absolute noise differences by running 10 times with different random seeds, averaging and performing Gaussian smoothing on the averaged map. Finally, we normalize the values of the smoothed map to  $[0,1]$  as the final attention map  $\mathcal{A}(\mathcal{I})$ . The pixel with larger value in the attention map should be added with the noise with larger variance, to sufficiently edit the content of the pixel. The pixel with smaller value in the attention map should be added with noise having smaller variance to preserve the content of pixel. We transform the attention map  $\mathcal{A}(\mathcal{I})$  into the weighting matrix  $\Lambda(\mathcal{I})$  through a Sigmoid function by  $\Lambda(\mathcal{I}) = \frac{1}{1 + \exp(-(a\mathcal{A}(\mathcal{I}) - b))}$ , where  $a$  and  $b$  are the hyper-parameters for the transformation. We discuss the effect of hyper-parameters  $a$  and  $b$  on the generation of images in Section 4.
In this section, we apply the proposed NGDM to image editing tasks, presenting both qualitative and quantitative results.
Evaluation tasks. We perform experiments on five datasets. First, we experiment on AFHQ dataset [28] to edit cats into dogs. The source domain contains 500 images of cats. These images are resized to  $256 \times 256$  resolution and subsequently edited into dogs. Second, we experiment on ImageNet dataset [29] to edit images from one class into another class based on text prompts, following the protocol of [30]. Third, we experiment on synthetic Imagen [31] dataset following DiffEdit [11]. We collect images generated by Imagen [31], along with their corresponding text prompts, and edit these images by altering portions of the text. Fourth, we experiment on COCO-S dataset and construct target prompts for editing from annotations provided by BISON [32]. We collect 1000 images and target prompts from the COCO [33] dataset to build the COCO-S dataset. We additionally consider DreamBooth dataset provided by [24], which contains 30 objects. Each object has 25 prompts and 4-6 images for editing. We edit each image using the provided 25 prompts, resulting in a total of 3950 edited images. More details are available in Appendix B.1.
Implementation details. We conduct experiments utilizing two types of pre-trained diffusion models. For the cats to dogs editing task on AFHQ dataset, we utilize the public pre-trained score-based diffusion model with the official code provided in ILVR [10]. This model operates directly in the image space. We set the denoising step  $N$  to 1000. We implement the remaining task based on the text-to-image latent diffusion model, i.e., Stable Diffusion [9]. This model was pre-trained on 512
$\times 512$  images of LAION dataset [34] with a latent dimension of  $64 \times 64$ . We use 50 steps in DDIM sampling with a fixed noise schedule. For hyper-parameters  $a$  and  $b$ , we set them to 10.0 and 5.0 respectively to obtain all qualitative comparison results presented in this paper. We conduct additional analysis to investigate the effect of different values of hyper-parameters in the experimental results. Unless otherwise stated, we use the default parameters of the diffusion model during inference.
Figure 2: Qualitative comparison on AFHQ dataset.
Results on AFHQ dataset. We report the widely-used Frechet Inception Distance (FID) [35] for quantifying realism and SSIM [36] for quantifying faithfulness. The quantitative comparisons and qualitative results are presented in Table 1 and Figure 2, respectively. The images generated by
our method better preserves the contextual structure (e.g., pose) of cats while yielding realistic dog images. For example, Figure 2 shows that for images in columns 2-5 with complex backgrounds, our method can accurately keep the backgrounds unchanged while editing cats into dogs. Other methods either blur the backgrounds or fail to maintain the source image backgrounds correctly. This observation is further supported by the quantitative results in Table 1, where our method achieves the best results of FID and SSIM among the compared SBDM-based methods.
Table 1: Quantitative comparison on AFHQ dataset. All results are reported by repeating experiments 5 times.
Method	FID ↓	SSIM ↑
StarGAN v2 [28]	54.88 ± 1.01	0.27 ± 0.003
CUT [37]	76.21	0.601
ILVR [10]	74.37 ± 1.55	0.363 ± 0.001
SDEdit [13]	74.17 ± 1.01	0.423 ± 0.001
EGSDE [14]	65.82 ± 0.77	0.415 ± 0.001
SDDM [38]	62.29 ± 0.63	0.422 ± 0.001
Ours	61.39 ± 0.27	0.478 ± 0.001
ILVR [10] SDEdit [13] EGSDE [14] Ours Source
Source
Oystercatcher Flamingo
Broccoli ↓ Butternut squash
Convertible Ambulance
Custard apple  $\downarrow$  Lemon
Bolete
ral fung
Kite  
↓  
Bald eagle
Figure 3: Qualitative comparison on ImageNet dataset.
Source
beach  
↓  
mountai
fuzzy panda  
↓  
raccoon
cat ↓ a Int
mountain  
 $\downarrow$   
beach
sunglasseses  
↓  
hat
skateboarding  
↓  
riding a bike
SDEdit
Figure 4: Qualitative comparison on Imagen dataset.