SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Quantitative comparison on ImageNet (a),Imagen (b), COCO-S (c), and DreamBooth (d) datasets. We report LPIPS distance [42] measuring image fidelity and CLIPScore [43] for text alignment. A higher CLIPScore denotes better alignment with the text, while a lower LPIPS value suggests higher fidelity to the input image. We report our results respectively using the default parameters  $a = 10.0$ ,  $b = 5.0$ , and the parameters  $a = 10.0$ ,  $b = 6.0$ .
Results on ImageNet dataset. Figure 3 shows that our method performs well on images even with complex backgrounds. For instance, when editing "custard apple" into "lemon", our generated image successfully preserves the intricate details of the tree branches. DiffEdit [11] generates unnatural images with artifacts. DiffuseIT [39] generates unnatural images, while DDIB [40] can hardly maintain the content of the source image. Figure 5(a) shows that our method outperforms other methods by achieving a better trade-off between CLIPScore and LPIPS value.
Results on Imagen dataset. We present our qualitative and quantitative results in Figures 4 and 5(b), respectively. Figure 4 shows visual results including background replacement and object properties modification. We can see that our method can generate images with better visual quality compared with the other methods. For instance, our results can successfully preserve the foreground while replacing the "beach" in the background with "mountain", or vice versa.
Results on COCO-S dataset. From Figure 5(c), when  $a = 10.0$  and  $b = 6.0$ , our method achieves the highest CLIPScore 31.45, and the smallest LPIPS value 23.43 compared with all other methods. InstructPix2Pix [41] is able to obtain CLIPScore comparable to that of ours, but the LPIPS value of InstructPix2Pix [41] is worse. We provide qualitative comparison in Appendix B.2.
Results on DreamBooth dataset. Figure 5(d) shows the quantitative results on DreamBooth dataset. As can be observed, our method outperforms compared methods by achieving a better trade-off
between CLIPScore and LPIPS value. DDS [44] achieves the best CLIPScore of 20.17 with a worse LPIPS value of 33.26. Our method obtains CLIPScore comparable to that of DiffEdit with better LPIPS value. We provide qualitative comparison in Appendix B.2.
Table 2: User study results on AFHQ dataset.
ILVR [10]	SDEdit [13]	EGSDE [14]	Ours
11.5%	10.5%	12.5%	65.5%
Table 3: User study results on ImageNet,Imagen,COCO-S,and DreamBooth datasets.
SDEdit [13]	DiffEdit [11]	DDS [44]	EDICT [45]	InstructPix2Pix [41]	Ours
4.5%	10.0%	3.0%	4.5%	6.0%	72.0%
User study. We conduct two user studies on AFHQ dataset and the remaining datasets including ImageNet, Imagegen, COCO-S, and DreamBooth. For each user study, 40 participants are provided with 30 randomly selected source images and the corresponding generated results of different methods. The generated images of ours and the other methods are displayed randomly in order. Participants are suggested to select the image that better applies the requested edit while preserving most of the original image details. The percentages of votes for
Table 4: The performance on AFHQ dataset with varying values of  $a$  or  $b$  while respectively fixing  $b = {5.0}$  or  $a = {10.0}$  .
a (b = 5.0)	6.0	8.0	12.0	14.0
FID ↓	87.01	72.43	54.10	46.75
SSIM ↑	0.556	0.513	0.449	0.425
b (a = 10.0)	3.0	4.0	6.0	7.0
FID ↓	42.34	50.35	74.58	88.49
SSIM ↑	0.373	0.423	0.539	0.601
our method and the other method on different datasets are shown in Tables 2 and 3 respectively, demonstrating that the participants exhibit a strong preference towards our method.
Effect of hyper-parameters  $a$  and  $b$ . As mentioned in Section 3.4, we transform the attention map  $\mathcal{A}(\mathcal{I})$  into weighting matrix  $\Gamma (\mathcal{I})$  with hyper-parameters  $a$  and  $b$ . We control the initial time step of denoising each pixel by adjusting the hyperparameters  $a$  and  $b$ . In Table 4, we analyze the impact of hyper-parameters  $a$  and  $b$  on AFHQ dataset. The upper part of the table reports the results with varying  $a$  and fixed  $b = 5.0$ , and the lower part of the table shows results with varying  $b$  and fixed  $a = 10.0$ . With  $b$  held constant, a larger  $a$  results in higher level of noise to the image, leading to more accurate editing and obtaining smaller FID. A smaller  $a$  results in content preserving the original image, obtaining larger SSIM value. The behavior of  $b$  is opposite to  $a$ .
Comparison with hard weighting matrix. We compare with the strategy used to produce  $\Lambda (\mathcal{I})$  by  $\Lambda (\mathcal{I}) = \mathbb{I}(\mathcal{A}(\mathcal{I})\geq \eta)$ , where  $\eta$  is a threshold chosen in  $\{0.1,0.3,0.5,0.7,0.9\}$ . The generated weighting matrix by this strategy is a "hard" weighting matrix that only takes 0 or 1 entry. Our Algorithm 1 gradually increases the denoising region with the increase of the denoising steps in the diffusion. Each pixel begins to be denoised with the denoising time step according to its relevance to the editing task. This helps to avoid artifacts caused by a hard mask, as shown in Figure 7.
saaee
cat
sno
Figure 6: Failure examples. We show cases in which our method fails to generate high-quality edited results.
Failure examples We show several failure examples in Figure 6 that were unsuccessfully edited. This could be because the computed weighting matrix is not accurate for determining the scale of noise to be added to each pixel in the source image.
Image editing aims to modify a real image to generate the desired image, resulting in tasks including image translation [46], style transfer [47], inpainting [48], object modification [24], etc. We focus on image editing tasks that commonly require editing specific object/thing of an image while preserving the remaining parts of the image.
Figure 7: Edited images and heatmaps with soft and hard weighting matrix. The images in the second column represent the results generated by our method and the heatmap below the image depicts the weighting matrix  $\Lambda(\mathcal{I})$  defined in Section 3.1 in the paper. The images in columns 3-7 represent the results generated using the hard weighting matrix with threshold value in  $\{0.1, 0.3, 0.5, 0.7, 0.9\}$ . The heatmaps below the images represent the binary hard weighting matrix.
Diffusion models showcase remarkable results for image editing. SDEdit [13] employs the source noisy data as the starting point in the denoising process, and it explores the trade-off between realism and faithfulness by controlling the initialization of the denoising time. EGSDE [14] utilizes the energy functions trained on both source and target domains to guide the inference process. EGSDE uses the noisy data at  $0.5T$  time step as the starting point of denoising, where  $T$  denotes the total number of denoising time steps in DDPM. Compared with them, our derived sampling method incorporates an adaptive selection of the initial denoising time for each pixel during the denoising process. Methods of [12, 23, 24, 49] utilize text-conditioned diffusion models to fine-tune the text embedding of the object that needs to be preserved during editing using a few or a single image. DDS [44] utilizes delta scoring to provide effective gradients for editing. DiffEdit [11] uses the DDIM inversion method to obtain noisy data and automatically generates a binary mask to guide the denoising process. RePaint [50] tackles the inpainting task by taking the unmasked image region from the input image and the masked region from the DDPM generated image, using the hard 0-1 mask. Differently, we perform image editing by adding independent noise with different variances to different pixels, depending on a weighting coefficient matrix that contains soft weights. Pixels with less added noise will better preserve the content of the source image.
6 Conclusion, Limitations and Societal Impact
In this paper, we propose a Non-isotropic Gaussian Diffusion Model (NGDM) for image editing tasks. The NGDM adds independent Gaussian noises with varying variances to different image pixels. To avoid training score model for NGDM, we rectify NGDM into an isotropic Gaussian diffusion model and design a data sampling method for NGDM by using a pre-trained isotropic Gaussian diffusion model to generate images. We demonstrate that NGDM better trade-off the balance of realism and faithfulness than the state-of-the-art methods for image editing tasks.
A limitation of our method could be that incorrect weighting matrix may lead to the failure of the method. Moreover, our method relies on a pre-trained diffusion model. Artifacts are produced when the edit involves generation failure cases of the underlying model. In future work, we will design a better way to calculate the weighting matrix more precisely and efficiently. And we will explore the application of our method in downstream tasks, such as domain adaptation.
In our experiments, all the considered datasets are open-sourced and publicly available. Our work aims to manipulate images with minimum effort. However, this method might be misused by faking images. We will take care to exploit the method to avoid the potential negative social impact and we will help research in identifying and preventing malicious editing.
This work was supported by National Key R&D Program 2021YFA1003002, and NSFC (12125104, U20B2075,11971373).
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[2] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurIPS, 2021.
[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[4] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
[5] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
[6] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[7] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[10] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
[11] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR, 2023.
[12] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
[13] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[14] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. In NeurIPS, 2022.
[15] Xiang Gu, Liwei Yang, Jian Sun, and Zongben Xu. Optimal transport-guided conditional score-based diffusion model. In NeurIPS, 2023.
[16] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurIPS, 2022.
[17] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Sahara, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
[18] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In ICLR, 2023.
[19] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped Langevin diffusion. In ICLR, 2022.
[20] Vikram Voleti, Christopher Pal, and Adam Oberman. Score-based denoising diffusion with non-isotropic gaussian noise models. In NeurIPS, 2022.
[21] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[44] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV, 2023.
[45] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In CVPR, 2023.
[46] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[47] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE Trans Vis Comput Graph, 26(11):3365-3385, 2019.
[48] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, 2018.
[49] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[50] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.