DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
Authors: Chong Mou
Xintao Wang
Jiechong Song
Ying Shan
Jian Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: Diffusion model
Image editing
Image generation
Primary area: generative models
TL;DR: A tuning-free diffusion method for general and drag-style image editing.

Abstract: Despite the ability of text-to-image (T2I) diffusion models to generate high-quality images, transferring this ability to accurate image editing remains a challenge. In this paper, we propose a novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models. Specifically, we treat image editing as the change of feature correspondence in a pre-trained diffusion model. By leveraging feature correspondence, we develop energy functions that align with the editing target, transforming image editing operations into gradient guidance. Based on this guidance approach, we also construct multi-scale guidance that considers both semantic and geometric alignment. Furthermore, we incorporate a visual cross-attention strategy based on a memory bank design to ensure consistency between the edited result and original image. Benefiting from these efficient designs, all content editing and consistency operations come from the feature correspondence without extra model fine-tuning. Extensive experiments demonstrate that our method has promising performance on various image editing tasks, including within a single image (e.g., object moving, resizing, and content dragging) or across images (e.g., appearance replacing and object pasting). Code is available at https://github.com/MC-E/DragonDiffusion.

Published as a conference paper at ICLR 2024
DRAGONDIFFUSION: ENABLING DRAG-STYLE MA- NIPULATION ON DIFFUSION MODELS
Chong Mou Xintao Wang² Jiechong Song'
Ying Shan Jian Zhang 1.3+
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University
ARC Lab, Teneent PCG
Peking University Shenzhen Graduate School- Aahool-Rabbitpre AIGC Joint Research Laboratory
feechongm, xintao.alpha] @ gomai.com, (songjiechong, zhangjian.sz, @ @pku.edu.cn
Cantinmus Appraranne Repbaring Ohhjer Moving & Residin Dmaging
Objject Prieling
Conteal Dragding
Coatiauras Moniing
Figure 1: The image editing tasks that our DragonDiffusion achieve without training.
ABSTRACT
Despite the ability of text-to-image (T2I) diffusion models to generate high- quality images, transferring this ability accurate image editing remains a chal- lenge. In this paper, we propose novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models. Specifically, we treat image editing as the change of feature correspondence in pre-trained diffusion model. By leveraging feature correspondence, we develop energy functions that align with the editing target, transforming image editing operations into gradi- ent guidance. Based on this guidance approach, we also construct multi-scale guidance that considers both semantic and geometric alignment. Furthermore, we incorporate a visual cross-attention strategy based on a memory bank design to ensure consistency between the edited result and original image. Benefiting from these efficient designs, all content editing and consistency operations come from the feature correspondence without extra model fine-tuning. Extensive experi- ments demonstrate that our method has promising performance on various image editing tasks, including within a single image (e.g., object moving, resizing, and content dragging) or across images (e.g., appearance replacing and object pasting). Code is available at /rttps://githun Ctttttuu..cooo//ee////aaaooooo ffusion
INTRODUCTION
Thanks to the large-scale training data and huge computing power, generative models have devel- oped rapidly, especially text-to-image (T2I) diffusion models Saharia al. (2022); Rombach et al. (2022;; Nichol et al. (2022); Ramesh et al. (2022), which aims to generate images conditioned on given text/prompt. However, this generative capability usually diverse, and it is challenging to design suitable prompts to generate images consistent with what the user has in mind Mou et al. (2033 Zhang et al. (2023), let alone fine-grained image editing based on the text condition.
Corresponding author
This work supported by National Natural Science Foundation of China under Grant 62372016.

---

Published as a conference paper at ICLR 2024
In the community of image editing, previous methods usually designed based on GANs Abdal
et al. 2019 2020): Alaluf et al. (2022) due to the compact and editable latent space, e.g., the W
space in StyleGAN Karras et al. (2019) Recently, DragGAN Pan et al. (2023) proposes a point- to-point dragging scheme, which can achieve refined content dragging. However, it is limited by the capacity and generalization of GANs. Compared to GANs, diffusion model et al. (2020) has higher stability and superior generation quality. Due to the lack of a concise and editable latent space, numerous diffusion-based image editing methods
al. (2022); Feng et al. (2022); Balaji al. 2022) are built based on T2I diffusion models via correspondence between text and image features. Recently, self guidance Epstein et al. (2024) proposes a differentiable approach that employs cross- attention maps between text and image to locate and calculate the size of objects within images. Then, gradient guidance is utilized to edit these properties. However, the correspondence between text and image features is weak, heavily relying on the design of prompts. Moreover, in complex or multi-object scenarios, text struggles to build accurate correspondence with a specific object. In this paper, aim to investigate whether the diffusion model can achieve drag-style image editing, which is a fine- teee-gaaiedd and generalized editing ability not limited to point dragging.
In the large-scale T2I diffusion model, besides the correspondence between text features and inter- mediate image features, there is also a strong correspondence across image features. This character- istic is studied in Dang al. (2023), which demonstrates that this correspondence is high-level, enabling point-to-poin correspondence of relevant image content. Therefore, we are intrigued by the possibility of utilizing this strong correspondence across image features to achieve image edit- ing. In this paper, we regard image editing as the change of feature correspondence and convert it into gradient guidance via energy functions Dhariwal & Nichol (202T) in score-based diffusion [Sng et al. (2020b). Additionally, the content consistency between editing results and original images is also ensured by feature correspondence in a visual pross-attention design. Here, we notice that there is a concurrent work, DragDiffusion (Shi et al. (2023), studying this issue. It uses LORA Ryu (2023) to maintain consistency with the original image and optimizes the latent in specific diffusion step to perform point dragging. Unlike DragDiffusion, our image editing is achieved by energy functions and a visual Pross-attentio design, without extra model fine-tumin mee-tuning or new blocks. In addition, we can complete various drag-style image editing tasks beyond the point dragging, shown in Fig. 1
In summary, the contributions of this paper are as follows:
We achieve drag-style image editing via image feature correspondence in the pre-trained diffusion model. We also study the roles of the features in different layers and develop multi-scale guidance that considers both semantic and geometric correspondence.
We design a memory bank, further utilizing the image feature correspondence to maintain the consistency between editing results and original images. In conjunction with gradient guidance, our method allows direct transfer of T2I generation ability in diffusion models to image editing tasks without the need for extra model fine-tuning or new blocks.
Extensive experiments demonstrate that our method has promising performance in various image editing tasks, including editing within a single image (e.g.. object moving, resizing, and content dragging) or across images (e.g.. appearance replacing and object pasting).
2 RELATED WORK
2.1 DIFFUSION MODELS
Recently, diffusion models Ho ct al. (200)) have achieved great success in the community of image synthesis. It is designed based on thermodynamics SSobl-Dicsstein et al. (0155); Song & Ermon 2019), including a diffusion process and a reverse process. In the diffusion process, a natural image Xo is converted to Gaussian distribution Xy by adding random Gaussian noise with T iterations. The reverse process is to recover Xq from Xy by several denoising steps. Therefore, the diffusion model is train a denoiser, conditioned the current noisy image x, and time step t:
Ex.0teer~ [lle - (x 1)
where cg is the function of the denoiser. Recently, some cext-conditioned diffusion models (e.g., et al. (2022) are proposed. Especially
GLID Nichol et al. (2022) and (ubbleDiffusioonSD)RRRooooo SD, transforming x, the latent space 2t, significantly improves the generation performance. From
2

---

Published as a conference paper at ICLR 2024
the continuous perspective [Song al. (2020b), diffusion models can be viewed function (i.e., (x(,,,t) = xx, log g g(x))) )) that samples from the corresponding distribution Song & Ermon 2020) according Langevin dynamics Sick-Dickntein et al. (2013): Song & Ermon (2119)
2.2
ENERGY FUNCTION IN DIFFUSION MODEL
From the continuous perspective of score-based diffusion, the external condition y can be combined by conditional score function, i... Vx, log q(x, ly). to sample from more enriched distribution. The conditional score function can be further decomposed as:
[g( (gy(xx)) xx )x)(x)) x Vx, (xx) + Xx, log (((
Vx, log ((xely) Vx, log
(2)
((y)
where the first term is the unconditional denoiser, and the second term refers to the conditional gradient produced by an energy function y(x,t,y) g(xely). & can be selected based the generation target, such as a
& Nichol (2021) to specify the category of generation results. Energy function has been used in varous controlable generation tasks, c.g., sketch-guidee generation Voynoy et al. (2023), mask-guided generation Singh et al. (2023), universal guidance Yu et al. (2023); Bansal et al. (2023). and image editing Epstein al. (2024). These methods, inspire us to transform editing operations into conditional gradients, achieving fine-grained image editing.
2.3 IMAGE EDITING
In image editing, numerous previous methods Abbal et al. 2019 2020): ;
Alaluf al. (2022) invert
images into the latent space of StyleGAN Karras et al. (2019) and then edit the image by manipulat- ing latent vectors. Motivated by the success of diffusion model Ho et al. (2020), various diffusion- based image editing methods Avrahami et al. (2022); Hertz et al. (2022); Kawar et al. (2023); [Meng al.) (2021): Brooks al. (2023) are proposed. Most of them use text as the editing control. For example, Kawar et al. (2023); Valevski et al. 2023); K Kwon& & Yo (2022) perform model fine-tuning on single image and then generate the editing result by target text. Prompt2Prompt /Hrtz et al. (2022) achieves specific object editing by exchanging text-image attention maps. SDEdit Meng et al. (202T) performs image editing by adding noise to the original image and then denoising under new text conditions. InstructPix2Pix [Brooks et al.] (2023) finetunes the diffusion model with text as the editing instruction. Recently, Self-guidance Epstein et al. (2024) transforms image editing operations into gradients through the correspondence between text and image features. However, the correspondence between text and image is weak, unable to perform fine-grained editing. Re- cently, DragGAN Pan et al. 2023) presents point-to-point dragging scheme. Nevertheless, its editing quality and generalization are limited by GANs. How to utilize the high-quality and diverse generation ability of diffusion models for fine-grained image editing is still an open challenge.
3 METHOD
3.1 PRELIMINARY: HOW TO CONSTRUCT ENERGY FUNCTION IN DIFFUSION
Modeling an energy function Exx;; y) to produce the conditional gradient Vx, log ((y(x,) in Eq. 2 remains an open question. & measures the distance between x, and the condition y. Some methods|Daame Dhariwal & Nichol (20T)): Voynov et al. (2023); Zhao et al. (2022) train a time-dependent distance measuring function, e.g., a classifier Dhariwal & Nichol (2021) to predict the probability that x, belongs to category y. However, the training cost and annotation difficulty are intractable our image editing task. Some tuning-free methods Yu al. 2023); Bansal et al. (2033) propose using the clean image Xoje predicted at each time step & to replace x, for distance measuring, i.e., El E(x;; f, R t. Nevertheless, there is a bias between Xojz and Xp. and there is hardly suitable D for distance measuring in image editing tasks. Hence, the primary issue is whether we can circumvent the training requirement and construct an energy function to measure the distance between x, and the editing target. Recent work Tang et al. (2023) has shown that the feature corre- spondence in the diffusion UNet-denoiser cg is leve- enabling point-to- point-to-point correspondence measuring. Inspired by this characteristic, we propose reusing ep as a tuning-fii free energy function to transform image editing operations into the change of feature correspondence.
3

---

Published as a conference paper at ICLR 2024
Reference Image
Appearance Editing
Keter Viww
Byy
[KC5 KY IoO
Kquu youu
Exmoaporakeee
vPd
Memery Bank
Craliat
Ciilinns
Ky.
Kyp V1
Original Image
Editing
Devnder
DDIM Inversion
Sdf #laiion
U-Net Thenclor
Z
Ren
(T-1)
Figure 2:
Overview of our DragonDiffusion, containing memory bank and score-based gradient guidance on the pre-traie pre-trained SD Rombach et al. (2022) without extra training or modules.
3.2 OVERVIEW
The editing objective of our DragonDiftusion involves two issues: changing the content be edited and preserving unedited content. For example, if user wants to move the cup in an image, the generated result only needs to change the position of the cup, while the appearance of the cup and other unedited content should not change. An overview of our method is presented in Fig. 2 which built on the pre-trained SD Rombach et al. (2022) to support image editing with and without reference images. Since SD is latent diffusion model (LDM), we first encode the original image Xo into the latent space zo. which is then reversed to zr by DDIM inversion Song et al. (2020a). If the reference image xi exists, it will also be involved in the inversion to produce zr In this process, we store some intermediate features and latent each time step to build memory bank, which is used to provide guidance for subsequent image editing. In generation, we transform the information stored the memory bank into content editing and consistency guidance through two paths, i.e. visual cross-attention and gradient guidance. Both of these paths are built based on feature correspondence in the pre-trained SD, without extra model fine-tuning or new blocks.
3.3 DDIM INVERSION WITH MEMORY BANK
In our image editing process, the starting point zy, produced by DDIM inversion et al. (2020a), provide good generation prior to maintain consistency with the original image. However, relying solely on the final step of this approximate inversion can hardly provide accurate generation guidance. Therefore, we fully utilize the information in DDIM inversion by building memory bank store the latent 2!nu at each inversion step well corresponding keys K and values V in the self-attention module of the decoder within the UNet denoiser. Note that in some cross-image editing tasks (e.g., appearance replacing, object pasting). reference images are required. In these tasks, the memory bank needs to be doubled to store the information of the reference images. Here, we utilize z, Kref, and V to represent them. The information stored in the memory bank will provide more accurate guidance for the subsequent image editing process.
3.4 GRADIENT-GUIDDNCE- EDITING DESIGN
Zo
Inspired by classifier guidance Dhariwal & Nichol (202)) build energy functions to transform im- age editing operations into gradient guidance in dif- fusion sampling. An intuitive illustration is presented in Fig. 3 showing a continuous sampling space of the score-based diffusion [Song et al. (2020b). The sam- pling starting point 2r, obtained from DDIM inversion, will approximately return to the original point only ac- cording to the gradient/score predicted by the denoiser. After incorporating the gradient guidance generated by the energy function that matches the editing target, the additional guidance gradient will change the path to reach a sampling result that meets the editing target.
Guidance
Continuous
Sampling Space
Original Gradlent - Corrected Gradlent Figure 3: Illustration of continuous sampling space in score-based diffusion. Bright colors indicate areas where target data is densely dis- tributed. The orange and green paths respec- tively refer to the diffusion paths without and with external gradient guidance.
4

---

Published as a conference paper at ICLR 2024
3.4.1 ENERGY FUNCTION VIA FEATURE CORRESPONDENCE
In our DragonDiffusion, energy functions are designed to provide gradient guidance for image edit- ing, mainly including content editing and consistency terms. Specifically, at the t-th time step, we reuse the UNet denoiser co extract intermediate features F from the latent z the current time step. The same operation is used to extract guided features F from z in memory bank. Following DIFT Tang et al. (023).. , Fi and Fq come from intermediate features in the UNet decoder. The image editing operation is represented by two binary masks (i.e., m and m to locate the original content position and target dragging position, respectively. Therefore, the energy function is built by constraining the correspondence between these two regions in F and F Here, we utilize cosine distance c-s(-) (.) E [-1, 1] to measure the similarity and normalize it to [0, 1]:
Stocat (F
In gen pagd i gud) = 0.5 cos
Fgen ( [m3e), sg ag(Fg FF(F"d(m gud) 0.5, (3)
where sg(-) is the gradient clipping operation. Eq. mainly used for dense constraints on the spatial location of content. In addition, a global appearance similarity is defined as:
ΣF Σ men
Σ Frud mmddd
Sglobal (Fgen mgen
Foud
m = 0.5 cos
sgi
Σ moad
+0.5, (4)
which utilizes the mean of the features region as a global appearance representation. When we want to have fine control over the spatial position of an object or a rough global control over its appearance, we only need constrain the similarity in Eq. [ and Eq. [ to be as large as possible. Therefore, the energy function to produce editing guidance is defined as:
Eedit a + 3 S(F?"", mm Fqrdd mond)"
S E {Stocats Sglobat ).
(5)
where a and are two hyper- paper--arameters, which are set as 1 and 4, respectively. In addition to edit- ing, we hope the unedited content remains consistent with the original image. We use mask m shore to locate areas without editing. The similarity between the editing result and the original image in m share can also be calculated by the cosine similarity as Stocal (Ft m shore Fr m share) ) Therefore, the energy function to produce content consistency guidance defined as:
Econtent
(6)
(K + (toeal(Far mohore powd m share)
In addition to Eedit and Ecomtents an optional guidance term Eapt may need to be added in some tasks achieve the editing goal. Finally, the base energy function is defined as:
E = We Eedis We Econtent + W. Eopt:
where We, We. and Wp are typer-parameten to balance these guidance terms. They vary slightly in different editing tasks but fixed within the same task. Finally, regarding [moen m shaaee] as condition, the conditional score function in Eq, 2 can be written as:
(8) The conditional gradient V. vegen log g(y (z can be computed by V Vegen E, which will also mul- tiplies by learning rate 7). In experiments, we find that the gradient guidance in later diffusion generation steps hinders the generation of textures. Therefore, we only add gradient guidance in the first n steps of diffusion generation. Experientially, we set n = 30 50 sampling steps.
gen log qZZ ly) O V agen log ((2)0)) gen log q(y 2!""),
[m gen II
sharey
3.4.2 MULTI-SCALE FEATURE CORRESPONDANCE
The decoder of the UNet denoiser contains four blocks of different scales. DIFT Tang et al. (2023) finds that the second layer contains more semantic information, while the third layer contains more geometric information. We also studied the role of features from different layers in image editing tasks, as shown in Fig. 4 In the experiment,
Zς random Gaussian noise and set mes n as zeros matrixes. m share is set as ones matrix. this way, generation relies solely on content consistency guidance (i.e,, Eq. (6) restore image content. We can find that the guidance
5

---

Published as a conference paper at ICLR 2024
Original Image
Guided by layer=]
Guided by Rayee-2
Guided the tayen-3
Guided by layer-4
Guided by the layer-2&63
Figure 4: Illustration of using features from different layers as guidance to restore the original image.
Zy is randomly initialized. The generation is solely guided by content consistency guidance in Eq.
from the first layer is too high-level to reconstruct the original image accurately. The guidance from the fourth layer has weak feature correspondence, resulting in significant differences between the reconstructed and original images. The features from the second and third layers are more suitable to produce guidance signals, and each has own specialty. Concretely, the features the second layer contain more semantic information and can reconstruct images that are semantically similar to the original image but with some differences in content details. The features the third layer tend to express low-level characteristics, but they cannot provide effective supervision for high-level texture, resulting in blurry results. our design, we combine these two levels (i.e., high and low) of guidance by proposing multi-scale supervision approach. Specifically, guidance on the second and third layers. The reconstructed results in Fig. also demonstrate that this combination can balance the generation of low-level and high-level visual characteristics.
compute gradient
3.4.3 IMPLEMENTATION DETAILS FOR EACH APPLICATION
Object moving. In the task of object mov- ing, m and gud locate the same object in different spatial positions. share is the complement (C,) of the union (U) of meen and mud i.e., m share However, solely using the content editing and consistency guidance in Eq. 5 and Eq. 6 can lead to some issues, as shown in the second image of Fig. [5] Concretely, although the bread is moved according the editing sig- nal, some of the bread content is still pre- served in its original position in the generated result. This is because the energy function does not constrain the area where the moved object was initially located, causing inpainting to easily restore the original object. To rectify this issue, use the optional energy term (i.e., Eapt in Eq to constrain the inpainting content to be dissimilar to the moved object and similar to a predefined reference region. Here, we use = to locate the reference region and define m (plp E m and menn) to locate the inpainting region. Finally, Expt in this task is defined as:
m
mss
(m s
Original Image
Moving w/o Eppt
Moving w Eppt
Figure 5: Visualization of the effectiveness of in- painting guidance (Eopt) in the object moving task, presenting that Eopt guide the inpainting of the area where the object is initially located.
Expt a + 8- Sylobal (F
-=ff) Stoent (Fgen
Food Spr).
mP Foud
where wi weight parameter, set 2.5 in our implementation. The third image in Fig. 1 ssows that this design can effectively achieve the editing goal without impeachable artifact.
Object resizing. The score function in this task is the same the object moving, except that scale factor > 0 is added during feature extraction. Specifically, we use interpolation to transform gud and Fo to the target size, and then extract Fgud [ as the feature of the resized object. To locate the target object with the same size in paen" we resize gen with the same scale factor y. Then we extract a new gen of the original size from the center of the resized gen, Note that if 7 < 1, we use to pad the vacant area.
Appearance replacing. This task aims to replace the appearance between objects of the same category across images. Therefore, the capacity the memory bank needs to be doubled to store extra information from the image containing the reference appearance, i.e., zze/ K,e,, and vrer gen and gud respectively locate the editing object in the original image and the reference object

---

Published as a conference paper at ICLR 2024
in the reference image. m maaare is set as the complement of m Le., (m To constrain appearance, choose Sglobut mmen Fre .m in Eq.
This task has no need for Eopt.
Object pasting, Object pasting aims to paste an object from an image onto any position in another image. Al- though it can be completed by simple copy-paste it of- ten results in inconsistencies between the paste area and other areas due to differences in light and perspective, shown in Fig- 6 can be seen, the result obtained by copy-paste exists discontinuities, while the result gener- ated by our DragonDiffusion can achieve a more harmo- nized integration of the scene and the pasted object. In implementation, similar to the appearance replacing, the memory bank needs store information of the reference image, which contains the target object. m and mowd respectively mark the position of the object in the edited image and reference image. m share is set as Cu Cu(m (m**)).
Reference Image
Prigin Imag
Copy paste
Pragonpifrusie Diffusio
Point dragging. In this task, we want to drag image con- tent via several points, as DragGAN Pan et al. (2023). In this case, ne and madd locate neighboring areas cen- tered around the destination and starting points. Here, we extract a rectangular patch centered around each point as the neighboring area. Unlike the previous tasks, ehore is manually defined.
Figure 6: Visual comparison between our DragonDiffusion and direct copy-
paste in cross-image object pasting.
3.5 VISUAL CROSS-ATTENTION
As mentioned previously, two strategies are used to ensure the consistency between the editing result and the original image: (I) DDIM inversion initialize zr; (2) content consistency guidance in Eq. (6] However, it is still challenging to maintain high consistency. Inspired by the consistency preserving in some video and image editing works Wu al. (202)); Cao al. (2023): Wang et al. 2023). we design a visual cross-attention guidance. Instead of generating guidance information through independent inference branch, we reuse the intermediate features of the inversion process stored in the memory bank. Specifically, similar to the injection of text conditions in SD Rombach al. (2022). we replace the key and value in the self-attention module of the UNet decoder with the corresponding key and value collected by the memory bank in DDIM inversion. Note that in the appearance replacing and object pasting tasks, the memory bank stores two sets of keys and values from the original image (Koud, vgudd and the reference image (KTe), Vrr/). ). In this case, concatenate the sets of keys and values in the length dimension. The visual cross-attention at each time step is defined as follows. C refers to the concatenation operation.
Q: QueR K. K (Kand @K,") V, Vguud (V.nd (V.")
(10)
Aur( K,, V,) = softmax
Q.K va V..
4 EXPERIMENTS
In experiments, we use StableDifflssin- 1.5 Rombach et al. (2022) as the base model. The infer- ence adopts DDIM sampling with 50 steps, and we set the classifier-free guidance scale 5.
4.1 COMPARISONS
In this part, we compare DragonDiffusion with other methods various image editing tasks.
Content dragging. In this task, we compare our method with the recent UserControllableLT Endo 2022),, DragGAN Pan et al. (2023), and DragDiffusion Shi et al. (2023). We first present the time complexity of different methods in Tab. [ Specifically, We divide the time complexity of differ- ent methods into two parts, i.e., the preparing and inference stages. The preparing stage involves Diffusion/GAN inversion and model fine-tuning. The inference stage generates the editing result. The time complexity is tested one point dragging, with the image resolution being 512 512.
7

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative evaluation on face manipulation with 68 and 17 points. The accuracy is calculated by Euclidean distance between edited points and target points. The initial distance (i.e., 57.19 and 36.36) is the upper bound, without editing. FID Seitzer (2020) is utilized quantize the editing quality of different methods. The time complexity is computed on the 'I point' dragging.
Preparing complexity
Inference complexity
Unaligned face
17 Points.
68 Points.
FID
From 57.19
From 36.36
17/68 points
Usenontrrllablee..T
1.2s
0.05s
X
32.32
24.15
51.20/50.32
DragGAN
52.40s
6.71s
X
15.96
10.60
39.27/39.50
DragDiffussin
48.25s
19.71s
22.95
17.32
38.06/36.55
DragonDiifussonnoous)))
3.62s
15.93s
18.51
13.94
35.75/34.58
Original Face
Reference Face Useroontrollabee..I
DragGAN
DragDiffusion
Ours
68 Points
Content Dragging
exxamee
exxmppe
Pasting
"a cake
Self-guldance
Ours
1d Self-guldaric
Ours
wceden hable
Replacing
Self-guidance
Ours
Self-guidance
Ours
Moving
apple woodes table"
apple wooden
table
To dog play with mon! To ddog play with man"
Figure 7: Qualitative comparison between our DragonDiffusion and other methods in face manipu-
lation (target points are blue), object pasting, appearance replacing, and object moving.
The experiment conducted that our method is relatively efficient in the preparing stage, requiring only 3.62s to prepare 2r and memory bank. The inference complexity is also acceptable for diffusion generation.
NVIDIA A100 GPU with Float32 precision. The results present
Following DragGAN Pan et al. point manipulation with 17 and 68 points. The test is randomly formed by 800 aligned faces (2018) training set. Note that we do not set fixed regions for all
(202).. the performance evaluation is conducted on the face key-
from CelebA-HQ Karras et al. methods, due to the difficulty manually providing mask for each face. In addition accuracy, we also compute the FID Seitzer (2020) between face editing results and CelebA-HO training set represent the editing quality. The quantitative and qualitative comparison presented in Tab. and Fig. 7 respectively. One can see that our DragonDiffusion achieves promising results in edit- ing accuracy and content consistency. Although DragGAN achieves better editing accuracy, it has limitations in content consistency and robustness in areas outside faces (e.g., the headwear is dis- torted). The limitations of GAN-based DragGAN and UserControllableLT also exist in requiring alignment before editing, shown in Fig. It can be seen that if editing without alignment, the results of DragGAN will suffer from severe degradation. The aligmment operation is not friendly to editing goal, as it will change the original image content, e,g., filtering the background. In
8

---

Published as a conference paper at ICLR 2024
Original Image
Moving w/o Inversion Prior
Moving wio Visual Consistency guidanee Cross-atteation
Moving w/o Content
Full Implementation
Figure Effectiveness of different components our DragonDiffusion in the object moving task.
comparison, method has promising editing accuracy, and the generation prior from SD enables better robustness and generalization for different content. In this task, our method also has better performance than DragDiffusion. More results are shown in the appendix.
Other applications. For object pasting, we compare our method with Paint-by-exaane Yang al. (2023). For appearance replacing and object moving, compare our method with Self- Guidance Epstein et al. (2024). The visual comparison in Fig. 7 shows that our method can achieve comparable performance the training method (i.e., Compared to self-guidance, our method has better edit- ing accuracy and content consistency. Due the lack of consistency constraints, Seff-Guidance produces some unexpected artifacts. Moreover, Self-Guidance has ob- vious deviation in complex scenes, due to the coarse correspondence between text and image features. More results presented in Appendix.
in object pasting.
4.2 ABLATION STUDY
Original lingee
DragGAN wio alignment
DragGAN aligoment
Ours wio afigoment
In this part, we demonstrate the effectiveness of some components in our DragonDiffusion, as shown in Fig. 9 We conduct the experiment on the object mov- ing task. Specifically, (1) we verify the importance of the inversion prior by randomly initializing 2r instead of obtaining from DDIM inversion. As can be seen, the random Zr leads to significant difference between the editing result and the original image. (2) We remove the content consistency guidance (i.e., Econtent) in Eq. 7 which causes local distortion in the editing result, e.g., the finger twisted, (3) We remove the visual cross-attention. It can be seen that visual cross-attention plays important role in maintaining the consistency be- tween the edited object and the original object. Using a memory bank to provide K, and V. can show ablation study for memory bank. There-
Editing comparison between our DragonDiffusion and DragGAN Pan et al.) (2023) the unaligned body and face.
Figure
greatly reduce the additional cost. In Appendix, fore, these components work together on both edited and unedited content, forming the fine-grained image editing model DiffuuDiifuusin, which does not require extra training modules.
CONCLUSION
Despite the ability of existing large-scale text-to-image (T2I) diffusion models to generate high- quality images from detailed textual descriptions, they often lack the ability to precisely edit the generated real images. In this paper, we aim to develop a drag-style and general image editing scheme based on the strong correspondence of intermediate image features in the pre-trained diffu- sion model. To this end, we model image editing as the change of feature correspondence and design energy functions to transform the editing operations into gradient guidance. Based on the gradient guidance strategy, we also propose multi-scale guidance to consider both semantic and geometric alignment. Moreover, a visual cross-attention is added based on memory bank design, which can enhance the consistency between the original image and the editing result. Due to the reuse in- termediate information from the inversion process, this content consistency strategy almost has no additional cost. Extensive experiments demonstrate that our proposed DragonDiffusion can perform various image editing tasks, including object moving, resizing, appearance replacing, object pasting, and content dragging. At the same time, the complexity of our DragonDiffusion acceptable, and it does not require extra model fine-tuning or additional modules.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432-4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegantt: How to edit the embedded images? Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8296-8305, 2020.
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inver- sion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pp. 18511-18521 2022.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble expert denoisers. arXiv preprint arXiv:2211.01322. 2022.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 843-852, 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18922-1802, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22560-22570, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 111888008794, 2021.
Yuki Endo. User-controllable latent transformer for stylegan image layout editing. In Computer Graphics Forum, volume 41, pp. 395-406. Wiley Online Library, 2022.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. Advances in Neural Information Processing Systems, 36, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guid- ance for compositional text-to-image synthesis. In Proceedings of the International Conference on Learning Representations, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In Proceedings of the International Conference on Learning Representations, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. Proceedings of the International Conference on Learn- ing Representations, 2018.
Tero Karras, Samuli Laine. and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
10

---

Published as a conference paper at ICLR 2024
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text- bext-baed real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Gihyun Kwon and Jong Chul Ye. Diffusion basod image translation using disentangled style and content representation. In Proceedings of the International Conference on Learning Representa- lions, 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In Proceedings of the International Conference Learning Representations, 2021.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint urXiv:2302.08453, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, In International Conference on Machine Learning, pp. 16784-168044 2022.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point- point-based manipulation on the generative image mani- fold. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204006125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 10684-10695 2022.
Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 35:36479-36494, 2022.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch frtorch-fid August 2020. Version 0.3.0.
Yujun Shi, Chubui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdif- fusion: Harnessing diffusion models for interactive point-based image editing. arXiv preprint S.XVv:2306.144335 2023.
Jaskirat Singh, Stephen Gould, and Liang Zheng. High-fidelity guided image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5997-6006, 2023.
Jascha Sohl- Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In Pro- ceedings of the International Conference Learning Representations, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ennon. Improved techniques for training score-bace based generative models. Advances in neural information processing systems, 13:11238-1222 2020.

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference Learning Representations, 2020b.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan. Uni- tune: Text-driven image editing by fine tuning diffusion model on single image. ACM Trans- actions on Gruphics (TOG), -20, 2023.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Aketch-guided text-to-image diffusion mod- els. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao. Xinlong Wang, and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models, arXiv preprint arXiv:2303. 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for texi-to-video generation. arXiv preprint arXiv:2212.1156. 2022.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 1839 2023.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided conditional diffusion model. In Proceedings of the IEEF/CVF International Con- ference on Computer Vision, pp. 23174-23184, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings the IEEF/CVF International Conference Computer Vision, pp. 3836-3847, 2023.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde:
Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623 2022.
12


Reviews:
[{"review_id": "K4t0iStGBy", "reviewer": "Reviewer_vgvS", "summary": "This paper tackles a suite of image editing task (dragging editing in particular) via gradient guidance to drive the sampling from the inversed latent towards the editing target. To ensure consistency between the editing output and the original input, the KV values are cached to form a memory bank that retains the semantics of the original image. The gradient guidance formulation can deal with multiple tasks like dragging edit, object removal, object resizing, appearance replacing and object pasting. The method is mainly compared with prior dragging-based image editing approaches and shows improved quality. The results on other image tasks are also quite impressive.", "questions": "I'm glad to see more comprehensive comparison against more approaches as aforementioned.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- While gradient guidance has been explored extensively, using this idea as a general approach to accomplish multiple image editing tasks is cool. The qualitative results as shown in Figure 12 is stunning. And, all of these are achieved without the use of any auxiliary model.\n\n- Caching the memory bank for improved image information preservation is a useful technique.\n\n- It is welcome to report the detailed model inference time as shown in Table 1. \n\n- The proposed method works well for real images, while the editing on real images is usually challenging for many prior methods.", "weaknesses": "- First of all, the paper proposes to use gradient guidance sampling for a bunch of tasks, but the paper writing and the experiments mainly focus on dragging-based editing. This will narrow down the scope of the paper quite a lot. It is suggested to formulate the paper as a general solution and equally treat multiple tasks. \n\n- Also, the experiment is not thorough enough. It is suggested to conduct comparisons on other tasks besides dragging edit. For example, for object pasting, it is suggested to compare against the work \"paint by example\". For appearance replacement, it is suggested to compare against \"Diffusion Self-Guidance for Controllable Image Generation\" and \"Null-text Inversion\".\n\n- There are some typos in the paper. For example, Equation 2 is not correct. \n\n- There is no limitation analysis for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2mj6GAEzP", "reviewer": "Reviewer_bwVZ", "summary": "The paper presents a method enabling user 'dragging' style motion control image editing, similar to DragGAN but with diffusion models. To achieve this, DDIM inversion is performed first on the input image, while the intermediate features from the UNet are saved. During the forward image editing pass, starting from the DDIM-inverted noise, at each diffusion step, three terms are calculated: 1. A cosine similarity score between the dragging patch of the DDIM inversion features and current generation features to gain local consistency, 2. A cosine similarity score between the mean features of the dragging patch of the DDIM inversion features and current generation features to gain global appearance consistency, and 3. Similarity between the unchanged features. The final score gradient is calculated by perturbing the original gradient with the gradient of a weighted sum of these similarities constraints to zt.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The task of user-defined handles is challenging and well-motivated -- – supported by various applications shown in the paper.\n* Evaluations were done with reasonable metrics and against SOTA methods, and decent improvements can be observed, especially the efficiency compared with DragDiffusion. Nice qualitative results are shown.\n* The method has significantly less complexity comparing with prior works, but seems to work well.", "weaknesses": "* Compared with prior (and concurrent) works such as DragGAN and DragDiffusion, way too few samples are shown. The paper and supplementary do not present enough challenging and diverse qualitative samples and comparisons.\n* The ablation is a bit incomplete. E.g. it will be nice to see some ablations on the usefulness of S_global.\n* Some flickering still happens in the no-change areas, e.g. clouds in the sun example and background in the apple example. If this is because of the balance between different losses, some ablation could be very helpful.\n* The identity preservation also seems a bit off, e.g. patterns of the apple. However, I think it is a relatively minor issue as other works also cannot completely fix this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XpyXPgmHjY", "reviewer": "Reviewer_Fbvf", "summary": "This paper introduces DragonDiffusion, an image editing method that allows for Drag-style manipulation on Diffusion models. By utilizing feature correspondence, this approach transforms image editing into gradient guidance. It incorporates multi-scale guidance that takes into account both semantic and geometric alignment, as well as visual cross-attention for consistency. The proposed method demonstrates promising performance across a range of image editing tasks.", "questions": "as weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The energy motivation that originates from classifier guidance is interesting. It motivates the design of the energy function for correspondence in diffusion models.\n- The visualization figure vividly demonstrates the editing effect.", "weaknesses": "- The clarity of how the memory bank is meaningful is not evident in this draft. As the memory bank is proposed as a contribution, the authors should provide a more comprehensive ablation study, including both quantitative and qualitative analysis.\n- How the energy design makes it works is not clear, the authors should provide more details numerical studies.\n- The inference time is too slow, approximately 15.93 in Table 1, which makes the solution incomparable with dragGAN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IWaJDx3VqG", "reviewer": "Reviewer_dwdi", "summary": "In this paper, the authors propose a method of drag-style image manipulation with Diffusion Model. They addressed this through the guidance based on feature matching and also conducted comparisons with previous studies such as DragGAN and DragDiffusion. The effects of using layers of various scales were analyzed, and attempts were made to preserve the content of the original image using visual cross attention. In addition, various applications such as object moving, object resizing, appearance replacing, and object pasting were also demonstrated.", "questions": "- Just as selecting the feature layer or combining information from the layer is important, I know that at which diffusion time the guidance is given also has a significant impact. Were there any related experiments on this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The training time is short and FID score is better compared to DragGAN and DragDiffusion\n- Many applications are conducted like object moving, object resizing, appearance replacing, and object pasting", "weaknesses": "- The problem targeted by this paper is not clear. Therefore, it is unclear why diffusion feature matching, drag-style editing, memory bank, and visual cross-attention strategy were introduced, giving an incremental feel.\n- If the paper is focused on the problem of drag-style image manipulation, more experimental results should be presented. For example, it is unclear why the FID score is higher compared to DragGAN. There is no related ablation study for that part.\n- Despite the introduction of the visual cross-attention strategy, it feels like that the identity or content of the image is not sufficiently preserved.\n- The choices of hyper-parameters seems heuristic. The experiment from multiple combination of hyper-parameter set could be helpful to address this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S3Min56HN5", "reviewer": "Reviewer_8eDU", "summary": "This paper presents DragonDiffusion - a novel framework to enable drag-style image editing with diffusion models.\nTo this end, new techniques are presented, including (1) using an energy function to guide the editing and (2) a memory bank for editing consistency.\nQualitative and quantitative experiments show the merits of DragonDiffusion.", "questions": "As both DragonDiffusion and Self-Guidance [4] use (1) an energy function and (2) modify the attention layer to perform edits, could the author elaborate further on the differences between them? I also think it would be great if the author could compare them to Self-Guidance (as they can also resize objects, move objects, etc.).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of enabling more precise and interactive image editing with diffusion models is an attractive topic. I believe that this work will attract both academic and community interest.\n- Sufficient experiments have been conducted to compare with DragGAN-related methods, showing the merits of the methods (e.g., DragGAN can not edit based on a reference image).\n- The ablation study demonstrates the effectiveness of the framework design.\n- Generally, the paper is easy to follow.", "weaknesses": "1. My primary concern about this paper is **whether ICLR is a suitable venue**.  \nI believe this paper would be more fitting for a Computer Vision conference (e.g., CVPR, ICCV, ECCV, SIGGRAPH).  \nWhile I don't intend to downplay the contribution of this paper (in fact, I appreciate it), I find it challenging to identify a precise description for this paper within the context of ICLR.  \nPerhaps, \"representation learning for application in Computer Vision\", but given that there is no \"representation learning\" happening, I am not sure. Thus, my initial rating is \"marginally below the acceptance threshold\".\n\n1. My other complaints are mainly about *writing* (but it is not the main reasons for my decision). Authors can use it to improve paper' clarity.\n- In Abstract, \"... they often lack the ability to precisely edit the generated or real images.\" -> I think this should tone down to \"interactively\" as \"precisely\" might have a broad meaning (e.g., precise in terms of pose, shape, etc.). In a broader meaning of precision, I see existing works can also achieve \"precise\" image editing (e.g., ControlNet [1]).\n- Introduction, first paragraph: (Similar to above) While I see DragonDiffusion has clearly advanced in interactive image editing, I think it'd be more comprehensive to mention seminal works aiming to perform more precise image editing (e.g., [1-4]... you name it). Alternatively, authors can briefly discuss these works in Section 2.3.\n- Section 2.3, \"InstructPix2Pix retrain diffusion models..\" -> \"InstructPix2Pix finetunes diffusion models...\"\n- Section 2.3, \"However, text-guided image editing is coarse.\" Could you add a sentence explaining why \"coarse\" is a bad thing?\n- Section 2 though Section 3.1 all use $x_{T}$, then suddenly Section 3.2 uses $z_{T}$. As far as I understand, the authors intend to use Latent Diffusion (Stable Diffusion), which is $z_{T}$. Then, could authors revise Section 2.1 (and other related parts in Section 2-3), so it is made sure that we have mentioned $z_{T}$ before?\n\n**Reference**:  \n[1] Zhang et al., Adding Conditional Control to Text-to-Image Diffusion Models, ICCV, 2023.  \n[2] Couairon et al., DiffEdit: Diffusion-based semantic image editing with mask guidance, ICLR, 2023.  \n[3] Nguyen et al., Visual Instruction Inversion: Image Editing via Visual Prompting, NeurIPS, 2023.  \n[4] Epstein et al., Diffusion Self-Guidance for Controllable Image Generation, NeurIPS, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)