The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing
Authors: Shen Nie
Hanzhong Allan Guo
Cheng Lu
Yuhao Zhou
Chenyu Zheng
Chongxuan Li
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
Stochastic differential equations
Image dragging
Image editing
Score-based models
Primary area: generative models
TL;DR: SDE is superior and verstile in image editing and SDE-Drag is powerful to drag open-set images

Abstract: We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose \emph{SDE-Drag}  -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed \emph{DragBench}) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.  See the project page \url{https://ml-gsai.github.io/SDE-Drag-demo/} for the code and DragBench dataset.

Published as a conference paper at ICLR 2024
THE BLESSING OF RANDOMNESS: SDE BEATS ODE IN GENERAL DIFFUSION-BASH IMAGE EDITING
Shen 2 Hanzhong Allan Guo 12 Cheng Lu?, Yuhao Zhou?,
Chenyu Zheng!. Chongxuan Lii
GGaoling School of Artificial Intelligence, Renmin University of China, Beijing, China "Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China DDpartment of Computer Science and Technology, Tsinghua University, Beijing, China fnieshen, guchanzhong, (Lucheng. ic15, yuhaoz.cs, chenyu, 2heng666 wtt.gami......
edu.
ABSTRACT
We present unified probabilistic formulation for diffusion-based image editing, where latent variable edited task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a correspond- ing SDE or ODE for editing. In the formulation, prove that the Kullback- Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-inn imoe translation, where SDE shows a consistent and substantial im- provement. Moreover, we propose SDE SDE-Drag a simple yet effective method built upon the SDE formulation for point-based content dragging. We build challeng- ing benchmark (termed DragBench) with open-set natural, art, and A Al-generated images for evaluation. A user study DragBench indicates that SDE-Dras sig- nificantly outperforms our ODE baseline, existing diffusion-basse basod methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion- based editing methods. See the project page https: : ///mmmm// /ml-gsai. gi thub. io/ DDE-Drag-demo for the code and DragBench dataset.
INTRODUCTION
The primary objective of image editing tasks is to manipulate the content of given image in controlled manner without deviating from the data distribution. As a representative example, point- based dragging (Pan et al., 2023) provides frienfriendly way to manipulate the image content directly. Although the field has witnessed numerous endeavors exploring diverse editing meth- ods (Meng al., 2021; Rombach al., 2022; Su et al., 2022; Wu & De la Torre, 2022; Couairon et al., 2022; Hertz et al., 2022; Zhao et al., 2022; Shi et al., 2023; Mou et al., 2023a), leveraging (large-scale) pre-trained diffusion models (Sohl-Dickstein al., 2015; Ho al., 2020; Song al., 2020b) to achieve remarkable empirical performance, there remains an absence of genuine attempts to comprehend and expound upon this process from a probabilistic perspective.
To this end, we present unified probabilistic formulation (see Fig. la, left panel) for diffusion- based image editing in Sec. 3.1, encompassing wide range of existing work (see Tab. 1). In the formulation, the input image undergoes inversion or noise perturbation first, followed by ma- nipulation or domain transformation to generate a task-specific latent variable. Starting from it, stochastic differential equation (SDE) or a probability flow ordinary differential equation (ODE) is defined by a pretrained diffusion model. Notably, they differ from the ones used for sampling because the corresponding marginal distributions mismatch. In Sec. 3.2, we theoretically show the
"Work done during an internship at ShengShu, Beijing, China. 1 "Corrsponndence Chongxuan Li.

---

Published as a conference paper at ICLR 2024
Editing
SDE-Drag
Inversion Forward
Manipulation Transformation
Fins
X
Xo-Po
contractive, ODE
Rto Tio-Pe Pio
Add noise
Capy&Passe
Second stage
unqqual
H
meequal
H
Sampling
PPo
SDE ODE
PPo
(a) Technical contributions. The left and right panels illustrate the unified probabilisttc perspective for image editing in diffusion and how to manipulate the latent variable in SDE--Drag respectively. Best viewed in color.
(b) Results in Dragging. The top and bottom rows visualize the results of SDE-Drag on real images and fake ones produced Stable Diffusion (Rombach al., 2022; Podell al., 2023) and DALL-E 3 respectively. In particular, last example shows that SDE- Drag can drag miniature lightning into the mug, which mentioned the text description yet missing in the original sample from DALL-E See more in Appendix G.3.
Figure 1: Overview of the paper. (a) Technical contributions. (b) Visualization of SDE-Drag.
promise of the SDE formulation for general image editing: the Kullback-Leibler (KL) divergence between marginal distributions of the SDEs decreases as the time tends to zero while that of the ODEs remains the same.
Inspired by the analyses, provide the SDE counterparts for widely used ODE baselines in various tasks. particular, we propose a simple yet effective method (dubbed SDE-Drag) built upon the SDE formulation for point-based content dragging (Pan et al., 2023) in Sec. 4.1. Distinct from the prior work (Pan et al., 2023; Shi et al., 2023; Mou et al., 2023a), SDE-Drag manipulates the latent variable in straightforward copy-and-paste manner (see Fig. la, right panel) instead of performing optimization in the latent space. We further build a challenging benchmark (termed DragBench) with 100 natural, art, and Al-generated samples open-set for evaluation in Sec. 4.2.
In Sec. 5, we conduct extensive experiments on various tasks including inpainting, sarage-to-imare translation, and dragging, where the SDE counterparts show consistent and substantial improve- ment over the widely used baselines. Notably, SDE-Drag can not only deal with open-set images but also improve the aligmment between the prompt and sample from advanced Al-painting systems like Stable Diffusion and DALL-E 3 (see Fig, lb).
BACKGROUND
We present preliminaries on diffusion models, SDE and ODE samplers, and data reconstruction algorithms in this section and discuss other related work in detail in Appendix C.
2

---

Published as a conference paper at ICLR 2024
Diffusion models. Let zy denote D-dimensional random variable follows data distribution qu (0(2)). Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020b) introduce a forward process {xx (x))))( that gradually adds Gaussian noise Zo and defines:
(0r(Zz(z0) = >N
(1 at)
(1)
where O₁ is a function of t that decreases monotonically. Besides, axp = 1 and at is sufficiently small ensure (or( (x0) (0, I). We denote the distribution of æ as q: (.x))). ). It is equivalent to represent the forward process as the following stochastic disferentiall equation (SDE):
dx; f(t) + g(t)dw,
(2)
where w; is standard Wiener process. For simplicity, we consider the VP type SDE (Song et al., 2020b), i.e., f(t) dloggo and 9² 2dt
da, dlog ax (1 as). The forward process has an equiv- alent backward process referred to as the reverse-time SDE, which recovers qo (mo) from (r(r)))
dx, = [t) 9 (t)V æ, log q: (zx1) dt + g(t)dr;
(3)
where w, is standard Wiener process with backward time and Va, log qr (2rz) is the score function of gr (2x)). A diffusion model parameterizes the score functions by a neural network 80(2,, (), which can be equivalently reparametrized as a noise prediction model EØ co(x),,) t). Taking eø (x(,,) as an example, the reverse-time SDE is given by
g2(1) (u 11 dt + g())
dx. = f(t)axe+ + VI- az
(4)
We denote the marginal distribution of x, defined by Eq. (4) as (x,, 1 Insightfully, there exists a probability flow ordinary differential equation (ODE), whose marginal distribution for any given t matches pa (xx) (Song et al., 2020b). Formally, the ODE is defined as:
dæ; = [(()az + 2/1 co( 1) dl.
g²(()
(5)
Samplers. Samples can be generated from the diffusion model by discretizing the reverse SDE (i.e., Eq. (4)) (Ho et al., 2020; Bao et al.. 2022b: Lu et al., 2022c) or ODE (i.c.. Eq. (5)) (Song et al., 2020a; Lu et al., 2022b) from T O. The ODE solvers often outperform SDE solvers given few (c.g., 10 - 50) steps and therefore are popular in sampling and editing. In particular, as a representative and widely adopted approach, DDIM (Song et al., 2020a) produces samples as follows:
8t = Vai
Ix
V1 agCo C,Co(
+ /1 a oeoooooee 8) + 0,W,
Vas
where 8 > L, G. = 7/(11 04)/(1 ax) ag/ar, 1 = 0 and の. is a standard Gaussian noise. For fairness, we employ DDPM (Ho et al., 2020) as the SDE sampler in our experiment, which also given by Eq. (6) with = 1. We present the algorithms of both samplers in Appendix A.
Data reconstruction. To preserve relevant information about the input image, most editing methods require way to produce latent variable that can reconstruct the input. This can be easily imple- mented by ODE samplers because of its invertability. fact, taking DDIM as an example, the latent variable can be directly produced by Eq. (6) with < and 57 O. which is called DDIM inversion.
However, the SDE is nontrivial to invert because of the noise injected. The previous work (Wu & De la Torre, 2022) proposes to invert the forward process of DDPM as follows:
as
Ot w, w NO
x3
V
+
at
C. VI
a c0.0 / a; 07
W/ 0 It v
04
(8)
Va.
where s > f. Intuitively, it first adds noise to obtain latent variables from data in Eq. (7) and solves the "noise" w. to be added to reconstruct the data through Eq. (6). is easy to check that we plug in wi and x, into the DDPM sampler in Eq. (6) with 77 1, we can reconstruct æ; perfectly. Note that it can be extended all SDE solvers and we refer to the sampling process of SDE given w', to Cycle-SDE for coherence. We present more details and algorithms of both reconstruction processes, as well as preliminary experiments on their reconstruction ability in Appendix A.
We omit the dependence 0 in Pe (ar:) for simplicity.
3

---

Published as a conference paper at ICLR 2024
Table 1: Summary of prior work in unified formulation. 12I means the image-to-image transla- tion task. As for the editing type, AM means to add mask to combine the sampling result and noise input. CL and CP mean to change the label and prompt during sampling respectively. OL means to optimize the latent variable via gradient descent. See more details in Appendix B. In this paper, we add SDE counterparts for the ODE-based methods, as highlighted blue in the last column.
Task
Methods
37
Edit
degg
-> zo
New results
Inpainting Meng al. (2021); Zhao al. (2022)
Noise
AM
SDE
Inpainting
Rombach et al. (2022)
Noise
AM
ODE
Tab, 2
121
Couairon et al. (2022); Hertz et al. (2022)
ODE
CP
ODE
Fig. 2
121
Su al. (2022)
ODE
CL
ODE
Tab. 6
121
Wu & la Torre (2022)
Noise
CL
Cycle-SDE
Dragging
Shi (2023); Mou al. (2023a)
ODE
OL
ODE
Fig. 3
3
THE BLESSING OF RANDOMNESS IN DIFFUSION-BASED IMAGE EDITING
In this section, we formulate a broad family of image-editing methods in diffusion models from a unified probabilistic perspective in Sec. 3.1 and show the potential benefits of SDE in Sec. 3.2.
3.1 A GENERAL PROBABILISTIC FORMULATION FOR IMAGE EDITING
In particular, given pretrained diffusion model parameterized by 0, an image Dy to be edited, and potential condition e such as a label or text description, we identify two common stages for our general probabilistic formulation.
The first stage initially produces an intermediate latent variable Tt, through either a noise-adding process (e.g.. Eq. (7)) or an inversion process (c.g., Eq. (6) with < t and = 0), where to E [0, is hyperparameter. Then, the latent variable Dt, is manipulated manually or transferred to different data domain by changing the condition e in a task-specific manner". We refer to the edited latent variable Etg. which deviates from the corresponding marginal distribution pt, (-) in general.
The second stage starts from Lt and produces the edited image žn following either an ODE solver (e.g.. Eq. (6) with 7) 0), SDE Solver (e.g.. Eq. (6) with r 1), a Cycle-SDE process (c.g., Eq. (6) with = and vi, from Eq. (8)). This formulation encompasses a board family of methods, summarized in Tab. 1. We illustrate these two stages in the left panel of Fig. Ia and please refer to Appendix B for more details.
Notably, the edited image æo does not follow po(-) in general because the marginal distributions at time to mismatch, making the editing process distinct from the well-studied sampling process. However, to the best of our knowledge, most of the prior work for image editing empirical and there remains an absence of attempts to comprehend and expound upon the editing process precisely. Moreover, as mentioned in Sec. 2 and Tab. 1, due to the popularity of ODE sampling and the property of easy to reverse, ODE is widely adopted in editing methods, and existing SDE-based approaches are rare and focus on a very specific task.
Our probabilistic perspective provides general and principled way to study the editing process, from which we attempt to answer the following natural and fundamental problems in the paper. Is there any guarantee the sample quality of the edited image (i.e., the gap between its distribution and the data distribution)? Do the SDE and ODE formulations make any difference in editing?
3.2 THEORETICAL ANALYSES
We present theoretical analyses here and we employ the widely adopted KI. divergence to measure the closeness of distributions by default. Throughout the paper, we focus on analyzing the different
'Heee we omit the detail that some existing methods manipulate the latent in multiple steps (see details in Appendix F). Nevertheless, our analyses in See. 3.2 apply both the one-step and multiple-step editing.
4

---

Published as a conference paper at ICLR 2024
behaviors of SDE and ODE formulations with mismatched prior distributions. Therefore, assume the model characterizes the true score functions (see Sec. 6) for simplicity. In such a context, all of ODE, SDE, and Cycle-SDE can recover the data distribution without editing. Formally, we assume that they characterize the same marginal distribution P: and p. gt for any E [0. 7). We present a discussion in Appendix D.1 about the rationality of using p as surrogate for & both theoretically and empirically. Besides, we consider general cases for all 0 < < t < T. By setting different values of s and t, our results apply both one-step and multiple-step editing. Please see Appendix for more details and all proofs.
We first prove that the KL divergence between the marginal distributions of two SDEs with mis- matched prior distributions gradually decreases while that for ODEs remains as the time approaches zero as shown by the following Theorems 3. 1-3.2, respectively.
Theorem 3.1 (Contraction of SDEs, Appendix D.3). Let P. and pt be the marginal distributions of two SDEs (see Eq. (4)) at time t. For any O S & < S T, if Pt. # Pt. then under some mild regularity conditions listed in Lu al. (2022a, Assumption A. it holds that
f. 9(T) Drisher (arshrr@rllpp (Frllp-)de dT DRI (pelpe))
(xa((ssllss)) = Dr.((ellpe))
where (-I-- denote the KL divergence and DFisher denote the Fisher divergence. Theorem 3.2 (Invariance of ODEs, see Appendix D.3). Let Pr and pt be the marginal distributions of two ODEs (see Eq. (5)) at time t. For any O S 8 <tts T, under some mild regularity conditions listed in Lu et al. (2022a, Assumption A. A. it holds that
DRt.(p.lpe3)
DEL. (pelpee)
(10)
Theorems 3.1-3.2 share the same spirit as Lyu (2012); Lu et al. (2022a). In particular, they follow from the Fokker-Plant equation and detailed computation of the time derivative of DKL (Pellpe).
Moreover, with a stronger yet standard assumption (i.e., the log-Sobolev inequality holds for the data distribution), we can obtain a linear convergence rate in the SDE formulation. Namely, we have DKL = exp(O(-(t -t-tttts)) (t)(tt-s))De DKL (Prllp)) We present the formal statement and more details in Appendix D.4 for completeness. Further, we conduct toy experiment on a one-dimensional Gaus- sian mixture data where the true score function has an analytic form and the log-Sobolev inequality holds illustrate the above results clearer in Appendix E.
The analysis of Cycle-SDE is more difficult because the random variables w', in the sampling path are correlated. Therefore, it is highly nontrivial to obtain a quantitative convergence rate for Cycle- SDE. However, we can still prove similar "contractive" result as in SDE using the data processing inequality for KL divergence (Cover, 1999). The result is presented in the following theorem.
Theorem 3.3 (Contraction of Cycle-SDEs, see Appendix D.5). Let Pe and p. be the marginal distri- butions of two SDEs (e.g., see Eq. (6) with r) = 1 and w. from Eq. (8)) at time t, respectively. For any 0 < s <t T. if # Pt. then
DkL(5.llps) < DKL
(11)
In summary, we show that the additional noise in the SDE formulation (including both the original SDE and Cycle-SDE) provides way to reduce the gap caused by mismatched prior distributions, while the gap remains invariant in the ODE formulation, suggesting the blessing of randomness in diffusion-basct image editing. Inspired by the theory, propose the SDE counterparts for representative ODE basclines across various editing tasks (as highlighted in blue in Tab. 1) and show consistent and substantial improvement in all settings (see Sec. 5).
4 DRAG YOUR DIFFUSION IN THE SDE FORMULATION
Among various image editing tasks, point-based dragging (Pan et al., 2023) provides user-frieet user-friendly way to manipulate the image content directly and has attracted more and more attention in diffusion models recently (Shi et al., 2023; Mou et al., 2023a). Inspired by the formulation and theory in Sec. 3, we investigate how to drag images in the SDE formulation (see Sec. 4. 4) and introduce a challenging benchmark with 100 open-set images for evaluation in Sec. 4.2.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 SDB-Drag
Require: Do. ag. ag; hyper-parameters r, m, n, to (0,T), (1, +00), B [0. 1]
for j in (1, 1,2.....77 m do
Obtain Tt [w/ according to Eq. (7-8) with ap, and to
Add and memorize noise
Obtain Tto according to Eq. (12) with o, 3, r and Tto
Manipulate the latent variable
Obtain To according Eq. (6) with r) L, {w) and Tto
Sample with Cycle-
Do f- ZD
Update the image
end for Return Zu
4.1 SDE-DRAG
We propose simple yet effective dragging algorithm based on the SDE formulation, dubbed SDE- Drag. We present SDE-Draag following the unified formulation Sec. 3.
In the first stage, we first add noise to obtain the intermediate representation It, and obtain series of "noise" w. according to Eqs. (7-8). Distinct from the prior work (Pan et al., 2023; Shi al., 2023; Mou et al., 2023a), we manipulate the latent variable in straightforward copy-and-paste manner (see Fig. la, right panel) instead of performing optimization in the latent space. In particular, given source point ag and a target point a provided by the user, let S and T denote the two squares with side length 2r centered at the two points, respectively, where r is a hyperparameter. Then, we first make a copy of atto Bto and modify Lt, follows:
3oo [T] aw [8] Tto [8] AT (3xr + VI B-E [8 ( N, I).
(12)
where a E (1, o.), , 3 E [0, 1) are hyperparameters, ax(S) takes pixels in æ that corresponds to the set S and \ is the complementary operation for sets. Intuitively, SDE-Draa directly copies the noisy pixels from the source to the target and amplifies them by a factor a. Besides, SDE-Drag perturbs the source with Gaussian noise by a factor / to avoid duplicating the content.
In the second stage, we directly produce žo by a Cycle-SDE process defined by Eq. (6) with v y 1, W. w. and ttoo tto. Besides, when an optional binary mask is provided, the unmasked area of the latent representation 2, at every step t is replaced by the corresponding unmasked area of the latent representation x in the noise-adding process. To our knowledge, this the first paper to investigate SDE formulations in dragging among prior work (Shi et al., 2023; Mou et al., 2023a).
When the target point is far from the source point, is challenging to drag the content in a single operation (Pan et al., 2023). To this end, we divide the process of Drag-SDE SDE into m steps along the segment joining the two points and each one moves an equal distance sequentially. We present the whole process in Algorihtm I with n discretization steps in sampling. and n are hyperparamters.
We emphasize that we employ default values of all hyperparameters (e.g., Or 3, m, and n) for all input images and SDE-Dras is not sensitive to the hyperparamettrss as detailed in Sec. 5.3. More- over, is easy to extend SDE-Drag by adding multiple source-target pairs either simultancously or sequentially. For more implementation details, please refer to Appendix F.6. Our experiments (see Sec. 5.3) on a benchmark introduced later demonstrate the effectiveness of the SDE-Drag over an ODE baseline and representative prior work (Pan et al., 2023; Shi et al., 2023).
4.2 DRAGBENCH
Built upon large-scale text-to-image diffusion models (Rombach et al., 2022), SDE-Drag and ODE- based diffusion (Shi et al., 2023; Mou et al., 2023a) can potentially deal with open- images, which is desirable. However, existing benchmarks (Pan et al., 2023) are restricted to specific domains like human faces. To this end, we introduce DragBench, a challenging benchmark consisting of 100 image-caption pairs from the internet that cover animals, plants, art, landscapes, and high-quality Al generated images. Each image has one or multiple pairs of source and target points and some images are associated with binary masks. Please refer to Appendix F.4 for more details.
Note that our analyses in Sec. 3.2 apply any time interval including this case.

---

Published as a conference paper at ICLR 2024
Table Results in inpainting. Inpaint-ODE and Inpaint-SDE employ DDIM (Song al., 2020a)
and DDPM (Ho et al., 2020) respectively. Inpaint-SDE outperforms Inpaint-ODE in all settings.
Small Mask (average mask ratio 0.2)
Large Mask (average mask ratio 0.4)
FID 4
LPIPS 4
FID 4
LPIPS +
# steps
25
50
100
25
50
100
25
50
100
25
50
100
Inpaint-ODE
5.84
5.86
5.86
0.227
0.227
0.227
17.73
17.57
17.50
0.363
0.361
0.361
Inpaint-SDE
5.18
4.97
0.218
0.215
0.215
16.43
15.48
15.27
0.351
0.345
0.343
5 EXPERIMENT
We conduct experiments on various tasks including inpainting (see Sec. 5.1), image-to-image (12I) translation (see Sec. 5.2), and dragging (see Sec. 5.3), where the SDE counterparts show a consistent and substantial improvement over the widely used baselines. We also analyze the sensitivity of hyperparameters and time efficiency (see Sec. 5.4).
5.1 INPAINTING
Setup. Inpainting aims to complete missing values based on observation. For broader interests, we follow the LDM (Rombach et al., 2022) to employ the Places (Zhou et al., 2017) dataset for evaluation and adopt FID (Heusel et al., 2017) and LPIPS (Zhang et al., 2018) to measure the sample quality. For simplicity and fairness, the base model (i.e., Stable Diffusson 1.5 (Rombach et al., 2022) trained on LAIOB-51 (Schuhmann et al., 2022)), and most of the hyperparameters remain the same the ODE baseline (Rombach et al., 2022). The only tuned hyperparameter is the number of sampling steps n and we conduct systematical experiments with 25, 50, and 100 steps. Please refer Appendix F.1 for more details.
As presented in Tab. 2, the SDE counterpart significantly outperforms the ODE baseline across all settings under both the FID and PLIPS metrics. Notably, with few steps (e.g.. 25), where ODE is widely recognized to have significant advantage in normal sampling, SDE still manages to achieve superior results. Further, the performance of SDE 25 steps surpasses that of ODE 100 steps, which strongly suggests the usage of SDE in inpainting. Additionally, we present the results of high-order algorithms in Tab. for completeness and the conclusion remains.
Results.
5.2 IMAGE-TO-IMAGE TRANSLATION
Setup. As visualized in Fig. 14 in Appendix 1.2, 12I aims to transfer the input images another domain by changing its label or text description. We provide the SDE counterparts for two represen- tative ODE approaches. Due to space limit, we present the results on DiffEdit (Couairon al., 2022) here and those on DDIB (Su et al., 2022) in Appendix G.2. For faireess, we use Stable Diffusion 1.5 (Rombach et al., 2022) trained on LAION-5B (Schuhmann et al., 2022)) and employ image- caption pairs from the COCO dataset (Lin et al., 2014) and annotations from the COCO-BISON dataset (Hu et al., 2019) following the DiffEEdit, Similarly, we adopt the same evaluation metrics in- cluding FID, LPIPS, and CLIP-Score (Radford et al., 2021). which measures the alignment between edited images and target prompts. As suggested by DiffEdit, we tune a single key hyperparame- i.e. the time index of the latent variable to. and keep others unchanged. The SDE counterpart (termed DiffEdit-SDE) employs the Cycle-SDE. Please refer to Appendix F.3 for more details.
Results. Quantitatively, DiffEdit-SDE outperforms the ODE baseline (termed DiffEdit-ODi ODEE) under all metrics with a wide range of In. as presented in Fig. 2. Qualitatively, as visualized in Fig. 14, DiffEdit-SDE achieves better image-text alignment and higher image fidelity simultaneously. We observe similar results in experiments with DDIB in Appendix G.2.
5.3 DRAGGING
Setup. As visualized in Fig. 17 in Appendix 1.4, dragging (Pan et al., 2023) aims to drag the content following the instructions of the source points and target points. We compare our SDE-Drag against
7

---

Published as a conference paper at ICLR 2024
16
0.26
Diffsdit-ODE Diff-dit-SDE
DiffEdit-OD ODE DiffEdit-SDE
14
0.24
FID
12
LPIPS
00.22
10
0.20
0.18
0.210
0,220
0,225
0.230
0.215
0.220
0.225
0.230
CLIP Score
CLIP Score 1
(a) Trade-off's between FID and CL.I--Score
(b) Trade-offs between I.PIPS and CL.P--coore
Figure 2: Results in 12I (DiffEdit). We consider to E {0.3, 0.5, 0.6, 0.7, 0.8). With the same value of to (linked by dashed lines), DiffEditSDEE outperforms DiffEdit-OD ODE under all metrics.
direct ODE baseline (termed ODE-Drag). ODE-based prior work (Shi et al., 2023), and the DragGAN (Pan et al., 2023). We focus on the challenging and desirable dragging task on open- set images and there is no standard benchmark in the literature far we know. Therefore, conduct a user study on the DragBench introduced in Sec. 4.2, as summarized below.
User Study. There were 6 participants and 100 questions for each vs. ODE-Drag) comparison by default. There were 22 questions to compare with DragGAN be- cause it is restricted to specific domains such as cats and lions. In each question, participants were presented with original images paired with two differently edited images produced by distinct mod- els. Participants were tasked with selecting a better image between the pair, which is known as the Two-Alterative Forced Choice methodology commonly used in the literature (Kawar et al., 2023; Bar-Tal et al., 2022; Kolkin al., 2019; Park al., 2020). See more details in Appendix F.5.
one (c.g.. SDE-Drag
Implementation. Throughout the paper, the hyperparameters in SDE-Drag are = 5, = 120, to 0.67, a = =111, = 0.3 and m = a. a. /2/ where l|a, - arll denotes the Euclidean distance between as and az If multiple points are present, m is determined based on the greatest distance among all pairs of points. To enjoy relatively high classifier-free guidance (CFG) and numerical stability simultaneously, we linearly increase the CFG from 1 to 3 as the time goes from 0 to lo. Note that SDE-Drag is insensitive to most of the hyperparameters, as detailed Sec. 5.4.
ODE-Drag shares all the hyperparameters as SDE-Drag except that it employs ODE inversion and sampling algorithms. Besides, following Shi et al. (2023), we integrate an optional LoRA (Hu et al., 2021) finetuning process for all iaffusion-based methods. Nevertheless, SDE-Drag without LoRA still outperforms ODE-Drag and DragGAN. See more implementation details Appendix F.6.
Results. As shown in Fig. 3, SDE- Drag significantly outperforms all competitors including the di- rect baseline ODE- -Dagg representative ODE- based prior work (Shi et al., 2023) and DragGAN (Pan al., 2023) on the challenging DragBench through comprehend user study. For completeness, also present the results without LoRA A against ODE-Drag and DragGAN in Fig. 7 in Appendix G.3 and the conclusion remains. In addition, the visualization results Appendix I.3 show that SDE- Drag can produce high-quality images edited in a desired manner, which agrees with the user study. All these results together with the ones in inpainting (Sec. 5.1) and 12I (Sec. 5.2) clearly demonstrate the superiority and versatility of SDE in image editing.
We further highlight that SDE-Drag is able to deal with multiple points simultaneously or sequen- tially on open-domain images and improve the alignment between the prompt and sample from the advanced Al Al-painting systems like DALL.E and Stable Diffusion (Podell et al., 2023) in Figs. 18- 20, advancing the area of interactive image generation.
5.4 SENSITIVITY ANALYSIS AND TIME EFFICIENCY
We perform a sensitivity analysis for all experiments. Overall, we did not tune the hyperparameters heavily and the SDE-based methods are not sensitive to most of the hyperparameters. In particular,
8

---

Published as a conference paper at ICLR 2024
100%
100%
100%
160
SDE-Dmg
SDE-Drag DragDaff
SDE-Drg DragGAN
SDE-Dmg
ODE Drag
140
7 DragANN
30%
30%
120
DragDifl
60%-
100
40%
40%
40%
20%
2075
2055
20-
0%
0%
066
10
(a) SDE vs. ODE
(b) Ours vs. DragDiff
(c) Ours vs. DragGAN
(d) Costing time
Figure 3: Results in dragging. (a-c) present the preference rates (with 95% confidence intervals) of SDE-Drag over ODE-Drag, DragDiffusion, and DragGAN. SDE-Drag significantly outperforms all competitors. The blank box in (c) denotes the ratio of the open-domain images in DragBench that DragGAN cannot edit. (d) shows that the average time cost per image is comparable for all methods.
Tab. 2 and Fig. 2 suggest that SDE-based methods can achieve good results in a wide range of key hyperparameters in inpainting and I2I. Below we focus the analysis of SDE-Drag where perform preliminary experiments on several images over a small set of values for hyperparameters.
Time and amplification factor We evaluated SDE-Drag with to/T" E {0.4, 0.5, 0.6, 0.7, 0.8). As identified in Meng et al. (2021) and confirmed in Fig. 9, a higher to improves the fidelity while lowering the faithfulness of the edited images and vice versa. However, the overall performance is similar for values between 0.5T and 0.7T and we set to = 0.67 by default. We also evaluate SDE- Drag with (1.0, 1.1, 1.2) and the default value of 1.1 performs slightly better (see Fig. 10).
Perturbation factor 3. We evaluate SDE-Drag with , E {0.1, 0.2, 0.3, 0.4, 0.5, 1}. Intuitively, when 3 1, the signal the source point is retained, leading to "copying" the content. For typical dragging, SDE-Drag is robust to values between 0.1 and 0.5. Fig. 11 visualizes the effect of B.
Dragging steps m. As discussed in Sec. 4.1, when the target point is far from the source point, it is challenging to drag the content with m = 1. As illustrated in Fig. 12, is sufficient to use an adaptive strategy with m = [[la, a.11/27. SDE-Drag robust when using a factor between 1/4 and 1 in the adaptive strategy.
LoRA. We perform systematical ablation study of LoRA in terms of user preference. We provide visualization results in Fig. 8 and more detailed analyses in Appendix F.6.
Notably, although the optimal hyperparameters may vary for each image, SDE-Drag works well in a wide range of hyperparameters and we provide a set of default hyperparameters for all images.
We now discuss the time efficiency. Overall, SDE-based methods achieve excellent results without increasing the computational time. As shown in Tab. 7 in Appendix G.4, the SDE counterpart takes nearly the same time as the direct ODE baseline in all experiments. Besides, SDE-Drag, DragDiffusion, and DragGAN have similar time efficiency as well, as shown in Figure 3d.
6 CONCLUSION AND DISCUSSION
We present a unified probabilistic formulation for diffusion- based image editing encompassing wide range of existing work. We theoretically show the promise the SDE formulation for gen- eral image editing. We propose simple yet effective dragging algorithm (SDF-Drag) based on the SDE formulation and a challenging benchmark with 100 open-set images for evaluation. Our results in inpainting, image-to-iman and versatility of SDE in general image editing.
translation, and dragging clearly demonstrate the superiority
There are several limitations. Theoretically, we do not consider the model approximation error and discretization error, which can potentially be addressed based on existing work (Lu et al., 2022a; Lee et al., 2022; 2023; Chen al., 2022: 2023). Besides, it is nontrivial to obtain convergence rate results for Cycle-SDE, which requires spsc-sc assumptions and analysis. Empirically, we observe that the open-set dragging problem is far from solved and present failure cases in Appendix J.

---

Published as a conference paper at ICLR 2024
Ethics Statement. This paper enhances the effectivennss of various image editing techniques, which leads to numerous societal benefits, including boosting productivity in visual industries, in- troducing modern tools educational curricula, and democratizing digital content creation for wider audience. However, it can also be harnessed to produce deceptive images or "deepfakes", and may give rise ethical concerns around image alterations that misrepresent reality history if not used properly. Besides, the editing process relies on pretrained diffusion model. If the model is trained on biased data, the editing process may amplify the bias.
ACKNOWLEDGEMENTS
This work was supported by NSF of China (No. 62076145); Beijing Outstanding Young Scien- tist Program (No. BJ/W2YJH01201110002200 Platform for the "Double-First Class" Initiative, Renmin University of China; the Fundamental Re- search Funds for the Central Universities, and the Research Funds of Renmin University of China (No. 22XNKJ13). C. Li was also sponsored by Beijing Nova Program (No. 20220484044).
Major Innovation & Planning Interdisciplinary
REFERENCES
Dominique Bakry, Ivan Gentil, Michel Ledoux, et al. Analysis and geometry of Markov diffusion operators, volume 103. Springer, 2014.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 2022.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.073000 07300, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang, Analytic-dpm: an analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint 2XXi:2201.0050: 2022b.
Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy- guided sde for inverse molecular design. arXiv preprint arXiv:2209. 15408, 2022c
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint 2035552303.006555 2023.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In European conference computer vision, pp. 707-723. Springer, 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation, arXiv preprint arXiv:2009.000 00710, 2020.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint 2221112009.1215. 2022.
Sitan Chen, Giannis Daras, and Alex Dimakis. Restorationnddegraaatton beyond linear diffusions: A non-asymptotic analysis for (dim-type samplers. In International Conference Machine Learning, pp. 4462-4484. PMLR, 2023.
Hyungjin Chung, Jeongsol Kim, Michael T Mecann, Mare L Klasky, and Jong Chul Ye. Diffusson posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210. /427, 2022.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
10

---

Published as a conference paper at ICLR 2024
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780- 34:8780-87744 87904,879444 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.016 01666, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. MrXiv:2207.125940 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6835 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. 2022.
Emiel Hoogeboom, Victor Gareia Satorras, Clément Vignac, and Max Welling. Equivariant diffu- sion for molecule generation in 3d. In International conference on machine learning, pp. 8867- 8887. PMLR, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00 006.00 2021.
Hexiang Hu, Ishan Misra, and Laurens Van Der Maaten. Evaluating text-to-image matching using binary image selection (bison). In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Inbar Huberman-Spiegellgaa Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint 204.v:2304.06140, 2023.
Alexia Joticoeur-Martineau, Ke Li, Rémi Piché-Taillefen Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2426-2435, 2022.
Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarrty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10051-10060, 2019.

---

Published as a conference paper at ICLR 2024
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint WxXiv:200.099611, 2020.
Holden Lee, Jianfeng Lu, and Yixin Tan, Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 25:22870-22882, 2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference Algorithmic Learning Theory, pp- 946-985. PMLR, 2023.
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Luping Liu, Yi Ren, Zhijic Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09773 2022.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. #4429-14460 PMLR, 2022a. URL httttt:///// /ttpp/ppocceeg..ee//e/p/p mlr prwsssppess//eeee / w1w2//1l222 html.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX00:221..01095, 2022c.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Siwei Lyu. Interpretation and generalization of score matching. arXiv preprint arXiv:1205.2622 2012.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.024211 2023a.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.084 08453, 2023b.
12

---

Published as a conference paper at ICLR 2024
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-88 8171. PMLR, 2021.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image mani- fold. arXiv preprini arXiv:2305. 10973, 2023.
Taesung Park, Jun- Yan Zhu, Oliver Wang. Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 33:7198-7211, 2020.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494, 2022.
André Schlichting. Poincaré and log- lob-lee inequalities for mixtures. Entropy, 21(1):89, 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25245 2022.
Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdif- fusion: Harnessing diffusion models for interactive point-based image editing. arXiv preprint arXiv:2306. 14435, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-vin video: Text-to-video generation without text-video data. arXiv preprint Sttiv:2209.1149900 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine leamn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456 2020b.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. arXiv preprint 22X8:22033088882 2022.
13

---

Published as a conference paper at ICLR 2024
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00493 2022.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, arXiv 2023.
preprint arXiv:2305.166
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Chen Henry Wu and Fernando De la Tore. Unifying diffusion models' latent space, with applica- tions to cyclediffusion and guidance. arXiv preprint arXiv:2210.05559. 2022.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li. and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. arXiv preprint arXiv:2309.05019 2023a.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation large mixture of diffusion paths. arXiv preprint arXiv:2305.18295, 2023b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Qinsheng Zhang. Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.00 05564, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6))1452-1444 2017.
14

---

Published as a conference paper at ICLR 2024
Algorithm 2 DDIM sampler
Require: 2T
1: for = 12 1,..., do
my; - 1-
2: Pt,-1 vat,11
+ v1
au, co(o,,
3: end for
4: return 270
Algorithm 3 DDPM sampler
Require: 2T
1: for = n + 1....,2 do
2: Tt,
ot 10111)((1 att)
as, 011-1
3: w; ~ 1),11)
14 (-- + v1- - 0e; =0
4: Tt yoa
(4) + oi, w.
v
5: end for
6: return 20
A BACKGROUND IN DETIAL
In this section, we detail the sampling algorithm (see Appendix A.1) and the data reconstruction methods (see Appendix A.2) employed our experiments. In addition, we present an empirical study of the reconstruction capability of Cycle-SDE and DDIM inversion in Appendix A.3.
A.1 SAMPLERS
We denote the noise prediction network as cø(0;, t) and define a time sequence [t creasing from t cally, when classifier-free guidance (Ho & Salimans, 2022) is employed, we have ca(2; 1) (0 1, Ø) + 8 (E,(
1n+1 in- and tn+1 T. where is the number of sampling steps. Specifi-
<0( t, 21). where s is the guidance scale, the conditional embedding and Ø is the unconditional embedding. We detail two representative ODE and SDE samplers, i.c., DDIM (Song et al., 2020a) and DDPM (Ho et al., 2020) in Algorithm and Algo- rithm 3, respectively.
A.2 DATA RECONSTRUCTION
With the same notation as Appendix A.1, below we summarize the data reconstruction methods based on ODE or SDE solver.
Due to the invertibility of ODE, given æp. we can deduce the latent representation Dr that ensures the reconstruction of æ via ODE sampling. Based on Eq. (5). the general formulation of ODE inversion is defined as:
1." ["[Ovee + g²(1) 2/1 at
co 1) dt,
VT = xo +
(13)
we detail the widely used DDIM inversion in Algorithm 4, which a special discretization method of general ODE inversion.
In the context of an SDE solver, given Tp, we first log a forward trajectory {x, 1n+1 em- ploying the forward precoss (i.e., Eq. (7)). Recall that each sampling step (e.g., Eq. (6) with
= 1) of the SDE solver (first order) can be denoted as Ft, f(eg, Zt, 4) + o w. where is function defined by sampling algorithm. Consequently, we can analyticly compute W; = (f(Eg, ft) æt...) /aa, based on Øt, and ot,., logged in the forward trajectory. With all {w,, 1n+1 calculated, we reconstruct Eg employ the SDE solver sampling from æy (i.e., Et.,,) ). We call these procedures Cycle-SDE and describe a DDPM-based Cycle-SDE Algo- rithm 5.
15

---

Published as a conference paper at ICLR 2024
Algorithm 4 DDIM inversion
Require: Zo
1: for = 2,... ++++0
vI (0 T-a;; ,60 -ii-1l)
2: Pt. Vaa,
Vou,
3: end for
4: return 2T
Algorithm Cycle-SDE (based on DDPM)
Require: 20 1: for i = -2,... + do
2: Dt, = -Var, at,, (01, 0t,-1 + V1 ar,/ar, w, w. N(0,1)
3: Tt. /11 a(, 1/(1 av.l / as, am
4: W de Ct: vaa, 21 I-a,, (o(
11-ai; of o0
yace;
5: Record ot. and wi
6: end for
7: for = 12 1....2. do
Sr. -1o,,, co(e,, V0z
8: Ct, var,
+ V
0t;; E0 (4) + or, wi
9: end for return Py. {-0
A.3 EMPIRICAL STUDY OF CYCLE-SDE
In this section, we discuss the reconstruction capability of Cycle-SDE, with classifier-free guidance (CFG) and compare it to the widely adopted ODE inversion method in image reconstruction.
In the procedure of Cycle-SDE, DDE, we first log forward trajectory {x,, yu+1 via the forward process, and the sampling process of Cycle-SDE reconstructs this trajectory through the analyticly com- puted ) In+1 (see Appendix A.2). However, practical constraints like floating-point precision can lead reconstruction errors. These eπrоrs especially significant when employing larger CFG scale, because the numerical instability inherent to CFG (Lu et al., 2022c). While the re- construction potential of Cycle-SDE has been touched upon in earlier research (Wu & De la Torre, 2022; Hluberman-Spiegellee et al., 2023), we present a complete discussion here.
Firstly, presented visual representation of Cycle-SDE's performance under various CFG scale and machine precision in Fig. 4. Cycle-SDE can flawlessly reconstruct the original image without CFG (i.e., CFG scale 1). However,
increase the CFG scale, such 4, the numerical instabil- ity intensifies, thereby hindering the ability to reconstruct the image. We demonstrate that exccuting experiments with double precision ensures stability and promotes successful image reconstruction, which shows that the primary source of Cycle-SDE's reconstruction error is numerical instability.
Furthermore, we also conducted quantitative experiments evaluate the image reconstruction ca- pability of Cycle-SDE and ODE inversion. The reconstruction upper bound was determined using the sector-quantized auto-encoder provided by Stable Diffusion (Rombach et al., 2022), denoted as VQAE. We employ Stable Diffusion 1.5 (Rombach et al., 2022) as our foundational model. Follow- ing Mokady et al. (2023), we randomly select 100 image-caption pairs from the COCO validation set for dataset and use PSNR to measure reconstruction quality. For the CFG, Following Mokady et al. (2023), we use a scale 7.5, which is also the default set in Stable Diffusion. We limited our sampling 50 steps for simplicity.
As outlined in Table 3, Cycle-SDe cannot successfully reconstruct the origin image when employing CFG. Implementing the experiment with double precision mitigates the numerical instability inher- ent to CFG. significantly reducing Cycle-SDE reconstruction error, and bringing it closer to the upper bound. Conversely, the reconstruction error in ODE inversion mainly due approximation
16

---

Published as a conference paper at ICLR 2024
Origin image
(b) CFG scale
(c) CFG scale 4
(d) CFG scale 4, double precision
Figure 4: Qualitative results in image reconstruction. Except for (d) which uses double precision, all other reconstruction experiments utilize single precision. Due to numerical instability in CFG, Cycle-SDe SDE struggles to reconstruct the original image while using double precision ensures numer- ical stability.
Table 3: Quantitative results in image reconstruction under CFG scale 7.5. For the experiment of Cycle-SDE, results from three trials were averaged due to randomness. Numerical instability in CFG affects both Cycle-SDE and DDIM inversion in reconstructing the original image. While dou- ble precision enhances stability, discretization errors in DDIM still dominate, preventing successful reconstruction.
Float32
Float64
VQAE
DDIM inversion Cycle-SDE
VQAE
DDIM inversion Cycle-SDE
PSNR 25.48
11.86
11.83
25.48
16.97
24.28
errors encountered during the ODE discretization phase. Therefore, enhancing machine precision has minimal impact on reducing the reconstruction error of ODE inversion.
B DIFFUSION-BASED EDITING METHODS
We present representative diffusion- editing methods mentioned in Tab. as instances of the general probabilistic formulation in detail.
17

---

Published as a conference paper at ICLR 2024
Inpainting aims to complete missing values based on observation. We start by discussing image sampling and then delve into the distinctions between inpainting and sampling. For a single sam- pling step in the diffusion model, either an SDE solver ODE solver is employed to sample from where < t. In the context of inpainting, SDEdit (Meng et al., 2021) and Stable Dif- fusion (Rombach et al., 2022) initially generate yt through the noise-adding process given the ref- erence image yo. and replace the unmasked area of 2; with the unmasked area of y: to produce Te. SDEdit and Stable Diffusion then sample from pe (Z. (ēx) employing an SDE solver and ODE solver, respectively. It is clear that Pto (-) * Pto (.) due to the manipulation.
DiffEdit (Couairon et al.. 2022) and Prompt-to- toopptto-Prompt (Hertz et al., 2022: Mokady al., 2023) are methods for image-to-image translation. In the first stage, both methods produce Lt, through ODE inversion employing the source image's ' prompt C1. For consistency with other methods, we denote Etg as t In the second stage, they utilize the target prompt 02 for ODE sampling from tto- Consequently, Ttoo ~ Pro (-Ic1) and mismatch exists between Pto (-(c) and Pto (-102).
DDIB (Su et al., 2022) introduces a distinct methodology for image-to-image translation. It employs a probabilistic approach similar to DiffEdit and Prompttt--Proppl The primary distinction is that DDIB uses class-conditional diffusion model, whereas DiffEdit uses a text-conditional diffusion model.
CycleDiffusion (Wu & De la Torre, 2022) is another method for image-to- tomtt translation tasks. CycleDiffusion produce Dt, through the noise-adding process and compute w', in E8. employing the source image's 's label 01- It then utilizes the target label 02 for Cycle-SDE sampling. Similar to DiffEdit and DDIB, P.o (l())= * Pto (-Ic2) in CycleDiffusion.
DragDiffusion (Shi et al., 2023) and DragonDiffusion (Mou et al., 2023a) are designed for image dragging. In the first stage, both methods produce Ttg through ODE inversion. After optimizing tt, with gradient descent, they produce tt, and then employ ODE sampling from #to- Clearly, we have Pro (+) # (-) due to the optimization procedure.
C RELATED WORK
Diffusion models. Diffusion models (Sohl-Dickstein et al., 2015; et., 2020; Song et al., 2020b) are able to generate high quality imags (Dhariwal & Nichol, 2021), audios (Chen et al., 2020; Kong et al., 2020), videos (Ho et al., 2022; Singer et al., 2022), point clouds (Luo & Hu, 2021), 3D (Poole et al. 2022: Wang et al., 2023), and molecular conformations (Hoogeboom et al., 2022; Bao et al., 2022c). Especially with the emergence of large-scale text-to-image models (Rombach et al., 2022; Ramesh et al., 2022; Saharia al., 2022; Bao al., 2023; Balaji et al., 2022; Xue et al., 2023b; Podell et al., 2023), there has been a significant advancement in the domain of image generation.
SDE and ODE Solvers. There are some works increasing the effectiveness of the diffusion model via solving reverse SDE or its equivalent reverse ODE with advanced methods. Ho et al. (2020); Song al. (2020b); Karras et al. (2022); Lu et al. (2022c); Bao et al. (2022b;a); Jolicoeur- Martineau et al. (2021); Xue et al. (2023a) introduce the reverse SDE discretization methods while Song et al. (2020a); Liu al. (2022); Lu al. (2022b;c): Zhang et al. (2022); Karras et al. (2022); Zhao et al. (2023) employ fast ODE sampling algorithm.
In comparison, previous work focuses on sampling from the diffusion model. In contrast, this paper studies SDE and ODE formulations in the context of image editing.
Diffusion-based image editing method. Based on the powerful open-set generation capabilities of the large-scale text-to-image diffusion model, the field of image editing has experienced rapid development. Meng et al. (2021): Zhao et al. (2022); Hertz et al. (2022); Wu De la Torre (2022); Couairon et al. (2022); Bar-Tal et al. (2022); Kawar et al. (2023); Kim et al. (2022); Mokady et al. (2023); Lugmayr et al. (2022); Wang et al. (2022); Chung et al. (2022) introduce advanced inpainting or image-to-image editing method employing various user control. We propose a general probabilis- tie formulation for image editing and analyze the difference between SDE and ODE when the prior distribution mismatch is due to manipulation or domain transformation during inference. Therefore, our formulation applics to all the above methods. However, training- based methods like Mou et al. (2023b); Zhang al. (2023) without mismatch, excluded from our framework.
18

---

Published as a conference paper at ICLR 2024
Inspired by the fast sampling of ODE solvers in image generation, ODE solvers are widely adopted in diffusion-buse image editing methods. Our theoretical and empirical evidence suggests that SDE is preferable in editing. Besides, existing SDE-based methods (Meng et al., 2021; Wu & De la Torre, 2022) focus on specific task but we demonstrate the superiority and versatility of SDE in general image editing.
Image dragging methods. Recently, Pan et al. (2023) introduced an interactive Point-based image editing method followed by Shi et al. (2023); Mou et al. (2023a). In comparison, Pan et al. (2023) is a GAN (Goodfellow et al., 2014; Karras et al., 2019) based method, implying its editing capabilities are confined to specific data, such as lions and cats. contrast, Shi al. (2023); Mou al. (2023a) are diffusion-bee based methods, possessing the capability for open-set editing. However, both of them employ ODE-based formulation, making it possible to further improve.
To our knowledge, SDE-Drag is the first SDE-based method for dragging open-set images. Besides, SDE-Drag manipulates the latent variable in simple and straightforward copy-and-passe paste manner instead of performing optimization in the latent space as in all prior work (Pan et al., 2023; Shi et al., 2023; Mou et al., 2023a).
D THEORETICAL ANALYSIS
D.1 THE RATIONALE TO MINIMIZE (Foolloo))
The final objective of image editing is analyze the editing distribution Po and data distribution do under certain divergence. In this section, we explain why making po and po close meaningful both theoretically and empirically.
Theoretically, we indeed assume that the diffusion model characterizes the true score functions, namely K L(pollgo) 0 in Sec. 3.2. Then our theory K L holds for K L /1oo) as desired. Intuitively, the assumption can be relaxed to a bounded approximation error of the score function, i.e., (xx,,
xx, In |el < e based on the latest theoretical work (Chen al., 2022: Lee et al., 2023; Chen et al., 2023). In particular, if e > 0, then the total variation distance (TV) for do and po tends to zero, namely, TV Po) - 0, making Po an accurate approximation for 40-
Empirically, experimental results suggest that Po is good surrogate for go. especially compared to Po- For instance, Stable Diffusion (Rombach et al., 2022) can generate high-fidelity images in human perception. Besides, Tab. 6 shows that the FID for sampling (namely po) is significantly lower than the FID for editing (ODE baseline, namely fo). suggesting that po is much closer to 40 than Pu. making it meaningful to minimize K LL(pollPo)
D.2 ASSUMPTIONS
Throughout this section, we adopt the regularity assumptions in Lu et al. (2022a, Assumption A.1). These assumptions are technical and guarantee the existence of the solution to ScoreSDEs, and make the integration by parts and the Tokker-Planck equations valid. For completeness, list these assumptions in this section.
For simplicity, in the Appendix sections, we use V--) to denote (.(.) and omit the subscript æ. And we denote V hx(x)= == tr( tr())(=)).
Assumption D.1. We make the same assumptions as Lu et al. (2022a, Assumption A.1) and we include them here only for completeness:
1. (o(x:) C and Exp(m) <c.
2. Vt E [0, T f(,t) E C And EC > 0, Vo E Rd, [0,7] llf(2 t)ll S C(1 + [|x1122.
3. 30 0, V2, y E Rd []f(xtt) fly. t)ll < C| 912.
4. g and Vt (0,T), lg(t)) 0.
5. For any open bounded set O, le So [=)01)) d. ((1))([[V (x)) co.
19

---

Published as a conference paper at ICLR 2024
6. 3C V, R i E [0, [V :
C(1 + |
7. 3C 0, y R | log (x:) V log q: (y)ll2 Cl vl12-
8. 3 Va Rd I. E [0 ||00 ar 1) - C(1 ||=|||))
9. 3C > 0, Væ, y E R | 1) 50 y 1)] - C||æ2 y122-
1 2 of V (()) so sa (x, t)]lddt) ] < cc.
10. Novikov's 's condition: E exp
11. Vt E [0.7],3k 0 (x()) = O ( pt (z) = O(e O(e-1lall)), [|all)),,, (x) =
O -o-l=l)) as |
D.3 ANALYSES OF SDE AND ODE
Theorem D.1 (Contraction of SDEs). Let pt and pr be the marginal distributions of two SDEs (see Eq. (4)) time respectively. For any S s < t T. if Pt + Pr. then
DKL. (psll=)) = Drel(Gellptt) - S. g(T)² Drisher (FrllP=)dr < DRI. (Pelpe),,
(14)
where D-L(-1)> denote the KL divergence and Drwher (-ll.) denote the Fisher divergence.
The proof shares the same spirit with previous works (Lyu, 2012; Lu et al., 2022a). In fact, the result a special case of Proposition C.I. Lu et al. (2022a). We add proof here for completeness.
Proof. We consider a genreal form of faz,t) and f(x,, 1) = f())a, in Eq. (2) is special case. The two reverse SDEs share the same score model 0 while starting from two different prior distri- butions pr and P. respectively. The process of both reverse SDEs is the same as follows:
dæ: [f t) - g(t)) (wr, t)]dt + g())
(15)
However, the induced marginal distributions and oe are different for any t E (0, because of the different priors, and by the Fokker-Planee equation, we have
(p)(x) Va (h(2, (x)) and Op.(#) Øt
V
(16)
Ot
where V. is the divergence operator, and
1
h(x,t) e f(x, g(t)) + g() V log);(x),
(17)
2
1
h( 1) 2 f( () - g(1)( 10(x, ((0)((((((()))))) (5+++19177 g0(() log
(18)
Expanding the time derivative of the KL divergence, obtain
0D (pel (p:)
pe (x(x))
(x( log (x)
dr
0t 0i
/ (1))
log (x()))
(x( 0p((.)
dx
(x()) Øt dx
Øt px(
/ V= (h(æ, ().(.)) log pí(x)) di
(h(æ, ()px
pt(
(r(
/ ((((a))))(())) V log
(x) dr / (1(22 V (x( dx (19)
(xx)
(x))
=f | [x())(((z))) - 1(0 t) [[[ (x) - Va logpi
1 50(1 / (e(()) (zg V log log
= Drisher (Fel.pt).
(20)
20

---

Published as a conference paper at ICLR 2024
Eq. (19) holds because of integration by parts with mild regularity assumptions (see Assumption A.I in Lu et al. (2022a)). Based on Eq. (20), the KL divergence between ps and ps is given by:
(Dr dr Ōr
DRL. (fsll))) DKI((fellp.)
[: g(7)² D(i))Drisheer (p> 11-111111 )dr
DRL(fellpe)
< <KuLfellp...
Theorem D.2 (Invariance of ODEs). Let Pt and Pt denote the marginal distributions of two ODEs (see Eq. (5) at time repectively. For any 0 S 8 < S T, it holds that
DKL(P,llp3 pps) (x.(p.llt).
(21)
The proof of Theroem D.2 is similar to Theroem D.1. In particular, it is easy to check that aDwr. llps) Øt = O, which is a special case Theorem 3.1 Lu et al. (2022a). We add proof here for completeness.
Proof. the ODE setting, ir and ÿt follow the Fokker-Planck equation Eq. (16) with h(x,() h(x t) = f) {t) (xx t). In view of Eq. (20), we know that
(Pkl(P.ll..) at
/ (x) h(xx al log Px(x) Va log 0.
Therefore, it holds that D (p.lpp))))) = (rilp))...
D.4 CONVERGENCE OF SDES
Following the main text, we have f(r, 2 l in the following analysis. Below intro- duce a widely used functional inequality, and we refer the interested readers to Bakry et al. (2014) for more details. We say distribution p satisfies the log-Sobolev inequality (LSI) if there exists (LS)(p) > 0 such that the following holds for every distribution q:
(xl(() < CLST(P)
E.g
V
log
(22)
2
and the smallest constant cps)(p) is called the log- -oo-lloeeev constant. Proposition D.I. Suppose that the LSI holds for the data distribution with CLST((a) 2 and Pt q: For any 1 1 T. Pt + Pt, then
DKL & exp (- CLSI(g.) [ (((())))))) (Pelle)
(23)
Proof. By Lee et al. (2022, Lemma E.7), we see that LSI also holds for or with E [s, and CLSi(gr) & max (CIsI(q:), 1} & cLsI(q=). From Eq. (20) and the PI, we know
-
Daa = g² (7)
V log
Ar Arr
g2(r) CLSI(g))
(f-1lt))
Jt
pr
which implies that
d S1 DEL (pr .lp))) 0.
dr
Therefore, the conclusion follows by integrating over r.
21

---

Published as a conference paper at ICLR 2024
D.5 ANALYSIS OF CYCLE-SDE
We first present a useful lemma.
Lemma D.1. Assume that for any arbitrary values of a, p(x) > ((<ly) > 0. If pp=((1))) y) = g( and p(y|z) q x) holds for all and for almost every we have p(a) - ((x)) for all x.
q(x) ((>ly) 0 and
q(0, y) Proof. By the definition of conditional distribution, for all and almost every y, have
p(a, y) p(y)
g(x, y)
and p(a,y) y)
( p(æ))
q(x))
which implies that
(()) - (())p(())
We finish the proof by taking integral y on both sides
p(x) = p(o))(y)dy = ((x))())d = q(x).
The SDE inversion algorithm does not follow the SamplingSDE process. However, it still reduces the KL divergence between two marginal distributions induced by processes from different prior distributions as the time approaches zero. This is formally characterized by the following theorem.
Theorem D.3 (Contraction of Cycle-SDEs). Let pt and Pt be the marginal distributions of two Cycle-SDEs (e.g., see Eq. (6) with = and w. from Eq. (8)) at time respectively. For any S S <t t T, if Pt + Pc, then
DkL(Pellp.) < (Pe(Pell...
(24)
Proof. We first introduce two joint distributions of æe and w. as follows:
(C a..) Pa (xz (x))) and (xr, x. |x)
where the conditional distribution (x, (x,) is defined by Eq. (6) with w. obtained from Algo- rithm 5. We denote the conditional distributions by Pt]s pt.a/ and pris Ptts/Ps.
Then, according the chain rule for KI. divergence Cover (1999), we have that:
DKL (f,s = DRL. (fe pt) + Ep, DKL. (Paje ll Pate)]
DKL. (ōs ps) + E,, [DRL. (Pu) pq+))]
20
which implies that DKL. (p.l(p.) & (felle)...). This is also known the data processing inequality of KI. divergence (Cover, 1999).
The remaining part of the proof shows that Ep. [DRL. (Peja |l Pq..]] 0 by contradiction. In fact, if Typ, [DKL (poje |l p 0, then P-j,(- Pate (ac.) holds almost surely, and according to Lemma D.1, we have pt Pt. which is contradiction. The assumptions in Lemma D.1 hold because of the added Gaussian noise in Algorithm
E TOY EXAMPLE
In this section, conduct toy simulation the Gaussian mixture data where the true score function has a closed form and the log-Soboler inequality holds to illustrate our theoretical results (Theorem 3.1, Theorem 3.2 and Proposition D.1) clearer.
Data distribution. We generate examples from a binary mixture of Gaussian distributions, that is, (x [ na, .²) |/ a²), where μ 0.5 and
0.2. Its score function has
22

---

Published as a conference paper at ICLR 2024
an analytic form, we derive it in the following for completeness. We know that 2, varzo + vI a where e, ~ (O 1). Then, by changing of variable of probability density, we have
1 N( Vosuu, + 1 Varzo 2
2
VI-are NN -a).
Combining these, we have
1 Valalvoill + ac)
(x())) 2 [ VaiH,, a,o² +1 ar) 2
- N(xelll - 0;) + [=N(xllue
2
(xt 154-110 ue)2
(xt + 14)
2 2/T00 exp(
exp(
207
2/2
207
Therefore, we can obtain the true score fuucctio as follows: s(x,, t) = V z: log qe (I²)
1
1
(xx, - 12)
34
H4
(x,, + 14,)2
It. + Ht. of
exp(
+
exp(
4: (xx))
207
of
2 2
20
exp(
(ir, (1.))
Rempie + exp(
(xxtt)))) 2
Botpe
2
exp(
(it p
+ expl
(scentrre))
207
207
(w+- )²
)2 (xotre)))
Pr exp(
- -exp(
20
20
of exp(
(at -1r²
+ exp(
(xx+t))) )2 20r
o
20²
Pt exp( 2W )
It
of
exp( 2x )
o7
M4Tt F4
tanh(
?
Fi
where we recall that 14g = Varft and of 0,00 2 Oz. In experiment, or is set by the cosine schedule in Nichol & Dhariwal (2021).
Besides, the log-Soboler inequality holds for go with CLSI(q0) S o (1+ (e (e" du² +1)) (see Schlichting (2019, Sec. 4.1) for details).
Sampler. Based on the true score function $y (z.). we can derive e(x,, t) - -I (2,, t) and adopt DDPM (i.e., Eq. 6 with 7 1) as the SDE sampler, DDIM(i.e, Eq. 6 with 77 0) as the ODE sampler. We sample data from different prior distributions including Gaussian distributions A( (2,, 2²) and and uniform distribution 21- 2].
Results and discussion. As shown in Fig. 5, when the prior distribution is (0, 1), getting benefit from the analytic score function s(x,, l), both SDE and ODE sampler recover go- However, when the prior distribution mismatch with the standard Gaussian distribution, the SDE sampler succeeds in recovering qo while the ODE sampler fails.
The simulation results illustrate our theory (Theorem 3.1, Theorem 3.2 and Proposition D.1) more clearly. On the one hand, Theorem 3.1 and Proposition D. state that DKL (PollPu) DKL ( (o) is exponentially smaller than DKL (Pr llrr), which means that though the prior mismatch, the SDE sampler can find Po that is similar to 90- However, in terms of the ODE sampler, Theorem 3.2 guarantees that Drr. (Pr (|p:) remains unchanged during the sampling process, which means that the ODE sample can never find Po that similar to 40-
F EXPERIMENTAL DETAILS
We present experimental details in this section.
23

---

Published as a conference paper at ICLR 2024
dats uls odic
odc
(a) (2r) (0
(b) pr (xr) (2,
(c) pr (TT) U -2
Figure 5: 1D toy experiment. (a) Both ODE and SDE samplers match the data if pr( = (0,,)) (b-c) ODE fails to recover the data distribution while SDE succeeds though the prior distribution mismatch with (œT) # 1).
F.1 INPAINTING
We first provide an overview the inpainting tasks and then detail the experimental settings of Section 5.1.
In the realm of image generation, ODEs converge more rapidly than SDEs (Song et al., 2020b;a; Lu al., 2022b;e; Karras al., 2022). As result, ODEs have been more prevalently utilized in inpainting tasks compared to SDEs. To our knowledge, we are the first to conduct a systematic comparison between SDEs and ODEs the context of inpainting, demonstrating that SDE can achieve superior results in inpainting tasks.
Inpainting- SDE and inpainting- ODE are equivalent the inpainting methods used by SDEdit and Stable Diffusion respectively, which are described in Appendix B.
We adopt Stable Diffusion 1.5 as our foundational model and consistently use the prompt "photograph" of a beauti full ful empty scene, highest quality settings" for all images following Rombach et al. (2022). For classifier-free guidance (Ho Salimans, 2022), we utilize commonly accepted scale of 7.5. In addition, we employ the mask provided by Li et al. (2022) in evaluation for simplicity.
F.2 DDIB
In this section, summarize DDIB (Su al., 2022) and detail the experimental settings of Ap- pendix G.2.
DDIB is magge-to-image translation method that leverages the reversibility of ODE. Please refer to Appendix B for more details about DDIB. To validate the superiority of SDE in general image editing methods, we replaced the ODE inversion and ODE solver in DDIB with the noise-adding and Cycle-SDE processes, respectively.
For evaluations, we utilize the FID measure the similarity between translated images and the target dataset, in addition to the SSIM and L2 metrics to gauge the resemblance between translated images and the source dataset. We set the scale of the classifier guidance to 1 (Dhariwal & Nichol, 2021), consistent with Su et al. (2022).
F.3 DIFFEDIT
In this section, we give an overview of DiffEdit (Couairon et al., 2022) and provide detailed experi- mental configurations of Section 5.2.
DiffEdit is a method designed for image-to-image translation. Given source image 2o accompa- nied by source prompt that describes it, and target prompt that outlines the desired editing image, DiffEdit first generates a mask M to highlight the editing area. Subsequently, the method uses the ODE inversion with the source prompt to obtain the latent representation It,, where serves as a hyper-parameter target prompt and the mask M to perform inpainting. For the general probabilistic formulation of
(0, T] DiffEddt then employs the ODE sampling from D,, employing the
24

---

Published as a conference paper at ICLR 2024
Table Categories in DragBench with corresponding image counts
Image type
animal
plant
natural landscape
Al-generated
Count
8
20
17
DiffEdit, please refer to Appendix B. In our SDE implementation, both the ODE inversion and the ODE solver are substituted by the noise-adding and the Cycle-SDE process, respectively.
adopt the evaluation pipeline from Couairon et al. (2022). For each image the COCO vali- dation set, COCO-BISON identifies captions that are not directly paired but are similar to the orig- inal. Consequently, we possess a source image with an accompanying source prompt and a target prompt paired with reference image for each edit. We utilize the Stable Diffusion 1.5 (Rombach et al., 2022) as our base model and apply the default classifier-free guidance scale 7.5 for both DiffEditSDDE and DiffEdit-ODE, The interval [0, 7] is discretized into 100 steps with varying to- For example, with to = 0.5T, both the inversion and sampling procedures consist of 50 steps each.
F.4 DRAGBENCH
In this section, we present DragBench, our newly introduced benchmark for image dragging. Drag- Bench consists of an open set of 100 images, spanning 5 distinct categories, as detailed in Table 4. Each image in the is accompanied by a textual description and at least one pair of source-tareet points. Furthermore, out of the entire collection, 44 images were randomly chosen. For these se- lected images, masks were applied regions that do not include the source-target points and visually should remain unchanged. We release DragBench on our project page.
F.5
USER STUDY
We provide more details about the user study of Section 5.3. Fig. 6 presents a screenshot of the interface in the user study, which includes the guidelines and a sample question provided to the participants.
F.6 DRAGGING
In this section, provide comprehensive overview of our proposed SDE-Drag and outline the implementation details of DragDiffusion and DragGAN.
We employ Stable Diffusion 1.5 (Rombach et al., 2022) the base model for both SDE-Drag and ODE-Drag. To enjoy relatively high classifier-fres guidance (CFG) and numerical stability simultaneously. we linearly increase the CFG scale from 1 to time goes from 0 to to.
Subsequently, we delve into the impact of LoRA fine-tuning. As depicted in Figs. (8a-8c), LoRA fine-tuning aids in retaining the core content of the image, particularly in arduous dragging tasks such as revealing the unseen facet of an object. Nonetheless, LoRA fine-tuning may result over- fitting the input image, compromising its editability, as evidenced in Figs. (8e-8g). To strike balance between preserving the core content and avoiding overfitting, we adopted a dynamic LoRA scale strategy. Specifically, we reduce the LoRA scale from to 0.5 time goes from tg- Note that a LoRA scale of () indicates reliance solely on the pre-trained U-Net weights, excluding LoRA parameters. Conversely, at a LoRA scale of 1, all LoRA parameters are utilized. At higher values of (e.g.. / 1o), the denoising process determines the object's outline. Thus, to circumvent overfitting, we opt for smaller LoRA scale. On the other hand, at lower values (e.g., / 0), the object's outline is already defined, and the denoising process merely augments details. Therefore, larger LoRA scale at this stage doesn't I pose a risk of overfitting We assessed the impact of reducing the LoRA scale from revealed minimal overall differences, leading us to settle on 0.5 as our choice. The influence of the dynamic LoRA scale is evident in Figs. (8d, 8h).
various values (0.7, O.5, 0.3, 0} across several images. Our observations
We set the fine- tuning learning rate at x 10 10-4, LoRA rank to 4, and training steps to 100. On a single A 100 GPU, this takes about 20 seconds for a single image.
25

---

Published as a conference paper at ICLR 2024
Annoration Guadiline
In wath 1et oe hawe twn imager witthed from the coms orggnn imaae udin
diferent mitreds
Reveorsse the oudin buttone sbown to ritrom the one tutter ectaed nonrr the reig
ltneare gradetmua
extting
Ohjectte in the orgiiall imagge shouid not chee For some images, we sell sino
meat, these abouad change dedior locationa
celeot adited before
Clicc
Figure 6: Screenshot from the User Study. The top and bottom row displays the guidelines and sample question provided to the participants, respectively.
For the baseline implementation, we follow the official code provided by DragGAN (Pan et al., 2023) and DragDiffusion (Shi et al., 2023) to edit the images in DragBench for user study and time comparison. For DragDiffusion, we stop optimization when all the source points are no more than pixel away from the corresponding target points and the maximum number of optimization steps is set to 40 following the official code. In addition, following DragGAN, we stop optimization when all the source points are more than d pixel away from the corresponding target points. When the number point pairs is no more than 5, d set to 1, otherwise is set to 2. The maximum number of optimization steps of DragGAN is set 300.
G ADDITIONAL RESULTS
G.1 ADDITIONAL RESULTS OF INPAINTING
In this section, to comprehensively demonstrate the superiority of SDE, we present inpainting results using the second-order ODE and SDE solvers. As shown Tab the SDE algorithm continues to outperform the ODE approach in the context of second-order solvers across all settings. However, when juxtaposed with the first-order solvers, the performance of the second-order algorithms falls short. We conjecture that this underperformance primarily stems from pixel manipulation magnify- ing the well-known instability tied to high-order algorithms (Lu al., 2022c).
G.2 ADDITIONAL RESULTS OF IMAGE-TO-IMAGE TRANSLATION
In this section, we present the results of another image-to-ing -omiggge translation method named DDIB (Su et al., 2022), illustrating the superiority SDE compared to the ODE baseline over the general image editing methods.
26

---

Published as a conference paper at ICLR 2024
Table 5: Results in inpainting under second order solver. Hipainting-ODE-2 and Inpainting- SDE-2 employ DPM-Solver++((MM and DDDE-DMM-Solver+<20 (Lu et al., 2022c) respectively. Inpaint-SDE. outperforms Inpaint-ODE-2 in all settings.
Small Mask
Large Mask
FID 4
LPIPS 4
FID +
LPIPS
# steps
25
50
100
25
50
100
25
50
100
25
50
100
6.12
6.03
5.95
0.229
0.228
0.227
18.13
17.82
17.67
0.363
0.362
0.361
Inpainting-SDDi SDE-2
6.11
5.39
5.10
0.224
0.219
0.217
18.04
16.20
15.61
0.355
0.347
0.344
Table 6: Results DDIB. Editing and Sampling represent image-to-image translation and image generation, respectively. SDE significantly outperforms ODE under FID in image-to-image transla- tion and exhibits more consistent performance between editing and sampling.
Lion + Tiger
Cock - Bird
FID
12
SSIM 1
FID |
12 J
SSIM f
Editing
DDIB-ODE
30.25 16.55
68.21
0.059 0.063
58,02 26.63
83.63 87.13
0.135 0.160
DDIB-SDE
67.83
Sampling
ODE 15.71
29.51
:
SDE 17.56
28.62
Setup. For fairness and consistency, we employ the ImageNet dataset and ADM (Dhariwal & Nichol, 2021) trained on ImageNet in the evaluation following DDIB. We adopt the wide-use metrics FID, L2 and SSIM (Wang al., 2004) in soage-to-image translation (Meng al., 2021; Zhao al., 2022; Wu & De la Torre, 2022) for quantitative analysis. We keep all hyperparameters the same as DDIB for simplicity and fairness. Furthermore, in order visually illustrate the difference between editing and sampling, we also conduct image-generation experiments on the target dataset, which is sampling from Gaussian noise that employs the target dataset label. For more discussion about the DDIB experiment please refer to Appendix F.2.
Results. As presented in Tab. 6, for translations from Lion to Tiger and Cock to Bird, the DDIB- SDE (i.e., the SDE-b DDIB) significantly outperforms DDIB-ODE (i.e., the ODE-based DDIB) in image fidelity, as gauged by the FID metric. While achieving high image fidelity, DDIB- SDE preserves similar image faithfulness to DDIB-ODE as measured by the L2 and SSIM metrics. Please refer to Fig. 13 for visualization results, in which DDIB-SDE shows higher image fidelity in image-to-image translation.
For ODE, there is significant performance difference between editing and sampling, with the edit- ing FID being notably inferior. Conversely, SDE demonstrates more uniform performance, with the editing and sampling FID scores closely aligned. This observation robustly validates motivation that SDE achieves superior consistency between editing and sampling.
G.3 ADDITIONAL RESULTS OF USER STUDY
In this section, we present additional user study results of SDE- SDE-Drag over ODE- and DragGAN without LoRA. As shown in Fig. 7, SDE--rrag consistently outperforms the two baselines.
G.4 ADDITIONAL RESULTS OF TIME EFFICIENCY
In this section, we discuss the time efficiency. As shown in Table 7, in all experiments, the SDE counterpart takes nearly the same time as the direct ODE baseline.
27

---

Published as a conference paper at ICLR 2024
100%
100%
SDE-Drag w/o LoRA
SDE-Drag LoRA DragGAN
ODE-Drag W/o LoRA
80%
80%
40%
20%
20%
I
0%
0%
(a) SDE w/o LoRA vs. ODE w/o LoRA
(b) SDE-Drag w/ LoRA vs. DragGAN
Figure 7: Results of dragging without LoRA. We present the preference rates (with 95% fidence intervals) of SDE-Drag over ODE-Drag and DragGAN without LoRA. SDE-Drag signif- icantly outperforms all competitors. The blank box in (b) denotes the ratio of the open-domain images in DragBench that DragGAN cannot edit.
Table 7: Additional results in per-image time costs. The SDE counterpart takes nearly the same time the direct ODE baseline.
Inpainting
DiffEdit
DDIB
Dragging
SDE
3.06s
4.82s
299.20s
75.86s
ODE
3. 3.11s
4.86s
300.18s
76.62s
Device
NVIDIA A100
NVIDIA A100
NVIDIA GeForce RTX 3090
NVIDIA A100
H HYPERPARAMETER ANALYSIS OF SDE-DRAG
We present more sensitivity analysis results of SDE- -rag in this section. Notably, SDE-Drag is not sensitive to most of the hyperparameters. To highlight the roles of individual parameters in SDE-Drag, we showcase images that respond differently to specific hyperparameter adjustments.
Fig. 8 shows the effect of finetuning LoRA. In addition, we display the analysis of hyperparameter to, a, and m Fig. Fig. 10, Fig. 11 and Fig. 12, respectively.
I VISUALIZATION RESULTS
I.1 DETAILS OF DDIB
As presented by Fig. 13, DDIB-SDE shows higher image fidelity in image-to-image translation compared with DDIB-ODE.
1.2 DETAILS DUFFEDIT
In Table 8, we present the quantitative metrics from the DiffEdit experiment, corresponding to Fig. in Section 5.2. Additionally, Fig. 14 offers visual comparison between DiffEdit-ShD and DiffEdit- ODE, demonstrating that DiffEdit-SDD SDE ensures superior image quality and better image-text align- ment.
28

---

Published as a conference paper at ICLR 2024
(a) Iaput image
(b) w/o fine-tuning LoRA (c) Constat LoRA scale 1 (d) Dynamic LoRA scale
(c) Input image
w/o fine-tuning LoRA (g) Constat LoRA scale (h) Dynamic LoRA scale
Figure 8: Analysis of LoRA fine-tuning, (a-c) demonstrate that LoRA fine-tuning aids in preserv- ing object consistency during editing, whereas (e-g) indicates a potential risk of overfitting to the input image. (d, h) illustrate how the dynamic LoRA scale strategy, as detailed in Appendix F.,, effectively balances between content retention and overfitting prevention.
(a) Input image
(b) to = 0.47
(c) to 0.67
(d) to = 0.87
Figure 9: Analysis of hyper-parameter ig- A lower to will lead to greater faithfulness to the input image but may compromise image quality. Conversely, a higher to enhances image fidelity the expense of faithfulness. However, in most of the cases, SDE-Drag robust to to between 0.57 and 0.7 T set the default to as 0.6 T.
I.3 DRAG COMPARE
In this section, we provide visual comparisons of SDE-Drag with DragDiffusion and DragGAN in Fig. 15 and Fig. 16, respectively.
1.4 DRAG FUN
In this section, we showcase the editing outcomes of SDE-Drag. The edits of real images are il- lustrated in Fig. 17. Edits of images generated by Stable Diffusion (Rombach et al., 2022; Podell et al., 2023) are shown in Fig. 18, and edits of DALL-E 3 generated images can be seen in Fig. 19. Moreover, highlight the methodology of dragging multiple points sequentially in Fig. 20.
29

---

Published as a conference paper at ICLR 2024
(a) Input image
a
a=)
(d) a =
(c) Input image
(f) a 1.
(g) a = 1.1
(h) 1.2
Figure 10: Analysis of hyper-parameter a. SDE-Drag is not sensitive to the value of a in most of the cases shown in the top row. However, observed that small number of images achieve better editing results when 1.1 as evidenced in bottom line. Therefore, we set the default a to 1.1.
(a) Input image
(b) 0.3
(c) 0=
3. For clarity, we fix the hhper--aramene parameter m to this figure. When 3 the object at the source point is retained, which can be used for "copying" objects. 3 value between 0.1 and 0.5 is suitable for "dragging" and produces nearly identical editing outcomes. Consequently, set the default 3 as 0.3.
Figure 11: Analysis of
FAILURE CASE
We show a failure case of SDE-Drag in Fig. 21 which indicates that effectively dragging open-set images is still significant challenge.
30

---

Published as a conference paper at ICLR 2024
(a) Input image
(b) [(s 1)
(c) m
Figure 12: Analysis of hyper-parameter
aa nz //2 represent that the distance in
each dargging operation is pixels. We found that pixel and 4 pixels per operation also work well. However, it is challenging when m 1.
(a) Source image
(b) DDIB-SDE
(c) DDIB-ODE
(d) Source image
(c) DDIB-SDI -DE
(f) DDIB-ODE
Figure 13: The visualization results of DDIB The source class is "lion", the target class is "tiger". DDIB-SDE achieves significantly better image fidelity compared DDIB-ODE.
31

---

Published as a conference paper at ICLR 2024
Table 8: Quantitative results of DiffEdit. For all evaluation metrics, DiftEdit-SDE consistently
surpasses DillEdit-ODE across different to-
Model
FID |
Clip-Score
LPIPS L
6g 0.3
DiffEdit-SDD
7.20
0.219
0.171
DiffEdit-ODS 7.67
0.215
0.173
tu 0.4
DiffEdit-SDE 7.29
0.224
0.181
DiffEdit-ODS
8.24
0.215
0.184
40- 0.5
DiffEdit-SDD SDE
7.57
0.228
0.195
DiffEdit-ODE
9.42
0.215
0.199
to = 0.6
DiffEdit-SDE 7.97
0.230
0.209
DiffEdit-ODE
11.27
0.214
0.214
fo = 0.7
DifiEdit-SDE 8.65
0.232
0.219
DiffEdit-ODE 13.51
0.212
0.228
lo- 0.8
DiffEdit-SDD DiffEdit-ODE
9.25
0.232
0.227
15.91
0.210
0.240
(a) Source image
(b) Obtained mask image
(c) DiffEdit- SDE
(d) DiftEdit-ODE
(c) Source image
(f) Obtained mask image
(g) DiffEdit-SDE
(h) DiftEdit-ODE
Figure 14:
Visualization results Diffedit. For the top row, the source prompt is bench near small stagrant lake" and the
"Young female sitting target prompt ""A man sitting on wooden bench near a lake." " For the bottom row, the source prompt is "A man sitting down at table using computer." " and the target prompt "An older man sitting at a desk looking at laptop. ". DiftEdit-SDE achieves significantly better image fidelity and image-text align- ment compared to DiffEdit-ODE.
32

---

Published as a conference paper at ICLR 2024
Figure 15: Visual comparison between DragDiffusion and SDE-Drag. The left column displays the original input images, while the center and right columns present edits from DragDiffusion and SDE-Drag, respectively.
33

---

Published as a conference paper at ICLR 2024
Figure 16: Visual comparison between DragGAN and SDE-Drag. The left column displays the original input images, while the center and right columns present edits from DragGAN and SDE- Drag, respectively.
34

---

Published as a conference paper at ICLR 2024
Figure 17: Editing results of SDE-Drag in real images.
35

---

Published as a conference paper at ICLR 2024
Figure 18: Editing results of SDE-Drag in images generated by Stable Diffusion.
Figure 19: Editing results of SDE-Drag in images generated by DALL-E 3.
36

---

Published as a conference paper at ICLR 2024
Figure 20: Results of dragging multiple points sequentially.
UNITI MABE
CHPY
EMIT WINE
Figure 21: Failure of SDE-Drag. For each input image shown the far left, tested SDE- Drag with three different random seeds. Such results suggest that dragging open-set images still significant challenge.
37


Reviews:
[{"review_id": "wy3epfpsSN", "reviewer": "Reviewer_NG3Y", "summary": "This paper show theoretically and experimentally the benefit of using diffusion model SDEs over ODEs for image editing. The authors formulate the image editing with diffusion model process in three steps : 1) encoding with deterministic or random noise 2) alteration of the latent which means modification of the prior distribution representing this latent 3) SDE or ODE sampling starting from the altered latent.\nOn the theoretical side, it is proven that during step 3, the KL distribution between the SDE marginals : a) when sampling from the altered latent and b) sampling from the original latent, decreases, while remaining constant when sampling with ODEs. On the experimental side, it is analyzed, from different works, the benefit of using SDEs over ODEs in step 3.", "questions": "- If the noise is fixed, is CycleSDE still an SDE ? \n- Is the log-Sobolev inequality likely to be verified in practice ?\n\nI am prepared to improve my score by taking into account the author's feedback.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The argumentation is limpid, and the contributions are clearly stated.  \n\nAlthough I am not an expert in image editing, I think that the main strength of the paper is its experimental study, which looks impressive. It contains numerous comparisons in three different tasks.  Moreover, the authors created an evaluation benchmark for point dragging and conducted a user study for evaluation on this problem. The advantage of SDEs over ODEs is clearly demonstrated experimentally.\n\nI like the fact that the authors took care to expose a very easy Gaussian toy example to illustrate the theorems.", "weaknesses": "Major weaknesses : \n- The problematic of the paper (end of Section 3.1) is never clearly answered. Actually, I do not think that the paper properly gives the answer to this question. This is link to the following point.\n- It is not clear that the proposed theoretical arguments prove the right point. Given $x_0$ (resp. $\\tilde x_0$) sampled from the latent $x_{t_0}$ (resp.  $\\tilde x_{t_0}$. ), with Theorem 3.1, it is proven that the $x_0$ and $\\tilde x_0$ are closer “in distribution” than $x_{t_0}$ and $\\tilde x_{t_0}$. Why is that desired ? \nI think that the authors see $p_0$ as the distribution of clean images $q_0$, and then wish to minimize $KL(p_0, \\tilde p_0)$ to get well-looking images. However, $p_0$ is very different from $q_0$ because the score is not (and for from being) perfectly matched with the denoiser. From this fact, could the authors explain why it makes sense to try to minimize $KL(p_0, \\tilde p_0)$ ? I am more likely to think that, for the purpose of image editing, it does not make sense to try to minimize the distance between these two distributions. \n- In the experimental section, some important information is missing : which model are you using, trained on which dataset ? \n\nMinor weaknesses : \n- In the paragraph \"Samples\" from Section 2. The term \"equivalent\" is not true. Sampling is not \"equivalent\" to discretization ! Moreover, discretizing (4) or (5) does not enable to sample from $q_0$ if the score is not perfectly matched with $\\epsilon_\\theta$.\n- The ODE inversion process explanation should be clarified. The links and differences between deterministic ODE inversion and random Gaussian noise should be explained. \n- The \"mild assumption\" should be detailed, at least in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QRUf5AU3cv", "reviewer": "Reviewer_UXKo", "summary": "This paper focuses on image editing using pre-trained diffusion models. The authors propose a unified probabilistic formulation for diffusion-based image editing, including inpainting, image-to-image translation, and dragging, based on existing methods. Experiments show that the proposed method achieves better performance than the original ODE version.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide SDE versions for existing ODE-based editing methods and achieve better performance.\n\n2. The authors build a benchmark called DragBench for evaluation, which may benefit the community.\n\n3. The paper is well written, and the experiment and presentation are solid.", "weaknesses": "1. The core idea that SDE beats ODE in image editing seems similar to CycleDiffusion [6]. This article seems to just generalize this phenomenon to multiple tasks and verify them. Overall, the innovative contribution seems insufficient. \n\n2. The authors claim to propose a unified probabilistic formulation for diffusion-based image editing. However, many related image editing/I2I methods are not mentioned, e.g., RePaint [1], DDNM [2], DPS [3], T2I-Adapter [4], and ControlNet [5].\n\nReferences:\n\n[1] Lugmayr et al., Repaint: inpainting using denoising diffusion probabilistic models, CVPR 22\n\n[2] Wang et al., Zero-shot image restoration using denoising diffusion null-space model, ICLR 23\n\n[3] Chung et al., Diffusion posterior sampling for general noisy inverse problems, ICLR 23\n\n[4] Mou et al., T2I-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models, arXiv 23\n\n[5] Zhang et al., Adding conditional control to text-to-image diffusion models, ICCV 23\n\n[6] Wu et al., A latent space of stochastic diffusion models for zero-shot image editing and guidance, ICCV 23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A6I4KwVQI8", "reviewer": "Reviewer_mBr1", "summary": "The paper presents a unified probabilistic formulation for diffusion-based image editing and introduces a simple yet effective dragging algorithm based on this formulation. The authors conduct experiments on various tasks, including inpainting, image-to-image translation, and dragging, demonstrating the superiority of their SDE-based approach.", "questions": "1. It is suggested to compare the performance of SDE in inpainting and image-to-image translation with the latest methods.\n2. Provide the running time of their proposed methods for inpainting and image-to-image translation tasks. And including the running time of the latest methods in these tasks to provide a more complete picture of the computational efficiency.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a comprehensive theoretical analysis of the SDE and ODE formulations for general image editing.\n2. The authors propose the SDE-Drag algorithm for dragging.\n3. The experiments across different tasks, including inpainting, image-to-image translation, and dragging, demonstrate the effectiveness of the SDE formulation in improving image editing tasks compared with the ODE baselines.\n4. The authors also provide the code, which shows the solidness of the work.", "weaknesses": "1. While the paper shows that SDE outperforms ODE baselines in inpainting and image-to-image translation, it lacks a comparison with the latest methods in these tasks.\n2. The paper provides the cost time for the dragging task but does not provide similar information for inpainting and image-to-image translation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)