SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
Authors: Chenlin Meng
Yutong He
Yang Song
Jiaming Song
Jiajun Wu
Jun-Yan Zhu
Stefano Ermon
Venue: ICLR 2022 Poster (2022)

Abstract: Guided image synthesis enables everyday users to create and edit photo-realistic images with minimum effort. The key challenge is balancing faithfulness to the user inputs (e.g., hand-drawn colored strokes) and realism of the synthesized images. Existing GAN-based methods attempt to achieve such balance using either conditional GANs or GAN inversions, which are challenging and often require additional training data or loss functions for individual applications. To address these issues, we introduce a new image synthesis and editing method, Stochastic Differential Editing (SDEdit), based on a diffusion model generative prior, which synthesizes realistic images by iteratively denoising through a stochastic differential equation (SDE). Given an input image with user guide in a form of manipulating RGB pixels, SDEdit first adds noise to the input, then subsequently denoises the resulting image through the SDE prior to increase its realism. SDEdit does not require task-specific training or inversions and can naturally achieve the balance between realism and faithfulness. SDEdit outperforms state-of-the-art GAN-based methods by up to 98.09% on realism and 91.72% on overall satisfaction scores, according to a human perception study, on multiple tasks, including stroke-based image synthesis and editing as well as image compositing.

Published as a conference paper at ICLR 2022
SDEDIT: GUIDED IMAGE SYNTHESIS AND EDITING WITH STOCHASTIC DIFFERENTIAL EQUATIONS
Chenlin Meng'
Yutong He
Yang Song'
Jiaming Song*
Jiajun Wu²
Jun-Yan Zhu
Stefano Ermon
Stanford University
*Carnegie Mellon University
ABSTRACT
Guided image synthesis enables everyday users to create and edit photo-realistic images with minimum effort. The key challenge is balancing faithfulness to the user inputs (e.g., hand-drawn colored strokes) and realism of the synthesized im- ages. Existing GAN-based methods attempt to achieve such balance using either conditional GANs GAN inversions, which are challenging and often require additional training data or loss functions for individual applications. To address these issues, introduce new image synthesis and editing method, Stochas- tic Differential Editing (SDEdit), based on a diffusion model generative prior, which synthesizes realistic images by iteratively denoising through stochastic differential equation (SDE). Given an input image with user guide in a form of manipulating RGB pixels, SDEdit first adds noise the input, then subsequently denoises the resulting image through the SDE prior to increase its realism. SDEdit does not require task-specific training or inversions and can naturally achieve the balance between realism and faithfulness. SDEdit outperforms state-of-the-art GAN-based methods by up to 98 09% on realism and 91. 11.72% on overall satisfac- tion scores, according to human perception study, multiple tasks, including ttroke-based image synthesis and editing as well as image compositing.
Stroke Painting Image
Stokk--hosed Editing
Input (guide)
Output
Image Compusiling
Source
Input (guido)
Output
Sornzz
Dorpet
Figure 1: Stochastic Differential Editing (SDEdit) is unified image synthesis and editing frame- work based on stochastic differential equations. SDEdit allows stroke painting to image, image compositing, and stroke-basee editing without task-specific model training and loss functions.
INTRODUCTION
Modern generative models can create photo photo-realistic images from random noise (Karras et al., 2019; Song et al., 2021), serving important tool for visual content creation. Of particular interest is guided image synthesis and editing, where user specifies a general guide (such as coarse colored strokes) and the generative model learns fill in the details (see Fig. 1). There two natural

---

Published as a conference paper at ICLR 2022
desiderata for guided image synthesis: the synthesized image should appear realistic as well as be faithfil to the user- user-guided input, thus enabling people with or without artistic expertise to produce photo-realistic images from different levels of details.
Existing methods often attempt to achieve such balance via two approaches. The first category leverages conditional GANs (Isola et al., 2017; Zhu et al., 2017), which learn direct mapping from original images to edited ones. Unfortunately, for each new editing task, these methods require data collection and model re-training, both of which could be expensive and time-consuming. The second category leverages GAN inversions (Zhu et al., 2016; Brock et al., 2017; Abdal et al., 2019; Gu al., 2020; Wu et al., 2021; Abdal al., 2020), where a pre-trained GAN is used to invert an input image to latent representation, which is subsequently modified to generate the edited image. This procedure involves manually designing loss functions and optimization procedures for different image editing tasks. Besides, it may sometimes fail to find latent code that faithfully represents the input (Bau et al., 2019b).
To balance realism and faithfulness while avoiding the previously mentioned challenges, we intro- duce SDEdit, a guided image synthesis and editing framework leveraging generative stochastic dif- ferential equations (SDEs; Song et al., 2021). Similar to the closely related diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020), SDE-based generative models smoothly convert an initial Gaussian noise vector to realistic image sample through iterative denoising, and have achieved unconditional image synthesis performance comparable to or better than that of GANs (Dhariwal & Nichol, 2021). The key intuition of SDEdit is "hijack" the generative process of SDE-based generative models, as illustrated in Fig. 2. Given an input image with user guidance input, such as stroke painting or an image with stroke edits, we can add a suitable amount of noise to smooth out undesirable artifacts and distortions (e.g., unnatural details at stroke pixels), while still preserving the overall structure of the input user guide. We then initialize the SDE with this noisy input, and progressively remove the noise to obtain denoised result that is both realistic and faithful to the user guidance input (see Fig. 2).
Unlike conditional GANs. SDEdit does not require collecting training images or user annotations for each new task; unlike GAN inversions, SDEdit does not require the design of additional training or task-specific loss functions. SDEdit only uses single pretrained SDE-based generative model trained on unlabeled data: given a user guide a form of manipulating RGB pixels, SDEdit adds Gaussian noise to the guide and then run the reverse SDE to synthesize images. SDEdit naturally finds a trade-off between realism and faithfulness: when we add more Gaussian noise and run the SDE for longer, the synthesized images are more realistic but less faithful. We can use this observa- tion to find the right balance between realism and faithfulness.
We demonstrate SDEdit on three applications: stroke-based image synthesis, stroke-based image editing, and image compositing. We show that SDEdit can produce realistic and faithful images from guides with various levels of fidelity. On stroke-based image synthesis experiments. SDEdit outperforms state-of-the-art GAN-based approaches by up to 98.09% on realism score and 91.72% on overall satisfaction score (measuring both realism and faithfulness) according to human judge- ments. On image compositing experiments, SDEdit achieves better faithfulness score and outper- forms the baselines by up to 83.73% on overall satisfaction score in user studies. Our code and models will be available upon publication.
2 BACKGROUND: IMAGE SYNTHESIS WITH STOCHASTIC DIFFERENTIAL EQUATIONS (SDEs)
Stochastic differential equations (SDEs) generalize ordinary differential equations (ODEs) by in- jecting random noise into the dynamics. The solution of an SDE is time-varying random variable (i.e., stochastic process), which we denote as x(t) E Rd. where
[0, 1] indexes time. In image synthesis (Song et al., 2021), we suppose that x(0) ~ Po = Paata represents a sample from the data distribution and that forward SDE produces x(f) for & E (0 via a Gaussian diffusion. Given
x x(), x(t) is distributed as a Gaussian distribution:
x(t) a(t)x(0) + at)z, z~ I(0,
(1)
2

---

Published as a conference paper at ICLR 2022
Perturb with
Reverse
Stroke
Stroke
Image
Image
Input
Output
Figure 2: Synthesizing images from strokes with SDEdit. The blue dots illustrate the editing pro- cess of our method. The green and blue contour plots represent the distributions of images and stroke paintings, respectively. Given a stroke painting, we first perturb it with Gaussian noise and progressively remove the noise by simulating the reverse SDE. This process gradually projects an unrealistic stroke painting to the manifold of natural images.
where (t) [0, 1] + [0, oo) is scalar function that describes the magnitude of the noise and a(t) [0. + jo, is a scalar function that denotes the magnitude of the data x(0). The probability density function of xx() is denoted Pt-
Two types of SDE are usually considered: the Variance Exploding SDE (VE-SDE) has a(t) = for all f and (1) being large constant that p1 is close (1))): whereas the Variance Preserving (VP) SDE satisfies a (t) + o² (t) = 1 for all f with a:(t) -> as t + so that p1 equals to (0, I). Both VE and VP SDE transform the data distribution to random Gaussian noise goes from 0 to 1. For brevity, we discuss the details based on VE-SDE for the remainder of the main text, and discuss the VP-SDE procedure in Appendix C. Though possessing slightly different forms and performing differently depending on the image domain, they share the same mathematical intuition.
Image synthesis with VE-SDE. Under these definitions, we can pose the image synthesis problem as gradually removing noise from a noisy observation x(() to recover x(0) This can performed via a reverse SDE (Anderson, 1982; Song et al., 2021) that travels from / = 1 to / = O, based on the knowledge about the noise-perturbed score function Vx log (x).. For example, the sampling procedure for VE-SDE is defined by the following (reverse) SDE:
dx(l) dlo²(1)) (x dt dl
dlo di
dw.
where w is a Wiener process when time flows backwards from conditions x(1) Pi N00, (1)I). then the solution x(0) will be distributed as Pdata In practice, the noise-perttee perturbed score function can be learned through denoising score matching (Vin- cent, 2011). Denote the learned score model so (x(t), t), the learning objective for time t is:
t==t=00 = 0. If we set the initial
L, =
Ilo, (x(()) 2|121.
where Pdasa is the data distribution and x(I) is defined in Equation The overall training ob- jective is a weighted sum over L of each individual learning objective Lt, and various weighting procedures have been diseussed in Ho al. (2020); Song et al. (2020; 2021).
With a parametrized score model sa (x(t), t) to approximate Vx log (x(), the SDE solution can be approximated with the Marer-Maruyama method; an update rule from (t + At) to is
x(t) x(t + At) + (d²(1) 02( (t))so(x(1))))+
oz -2(1)) At)z.
(4)
where z (O, We can select a particular discretization of the time interval from 0. initialize x(0) ~ 2 (1) and iterate via Equation 4 to produce an image x()).
GUIDED IMAGE SYNTHESIS AND EDITING WITH SDEDIT
In this section, we introduce SDEdit and describe how we can perform guided image synthesis and editing through SDE model pretrained on unlabeled images.
3

---

Published as a conference paper at ICLR 2022
Realidic
More falhful
naalitic faitffal
Less ralleeie
20020
SDEdit
15000
Faithful
Realistic
10000
Sweet spoc
04
0.2
0.4
0
= 0.2
0.4
0.5
0.6 0.7
to = 0.0
to 0.9
6=1
(a) KID and Le norm squared
(h) We illustrate synthesized images of SDEdit with various to initializations. to = 0 indicates the guide itself, whereas to indicates a random sample.
plot with respect to fo-
Figure 3: Trf between faithfulness and realism for stroke-based generation LSUN. As to increases, the generated images become more realistie while less faithful. Given an input, SDEdit aims at generating an image that is both faithful and realistic, which means that we should choose to appropriately (fo E [0.3, 0.6] in this example).
Setup. The user provides full resolution image x(9) in a form of manipulating RGB pixels, which we call a "guide". The guide may contain different levels of details; high-level guide contains only coarse colored strokes, a mid-level guide contains colored strokes on real image, and a low-level guide contains image patches on target image. We illustrate these guides in Fig. 1, which can be casily provided by non-experts. Our goal is to produce full resolution images with two desiderata:
Realism. The image should appear realistic (e.g., measured by humans or neural networks).
Faithfulness. The image should be similar to the guide x(9) (e.g., measured by Lg distance).
We note that realism and faithfulness are not positively correlated, since there can be realistic images that are not faithful (e.g., random realistic image) and faithful images that are not realistic (e.g. the guide itself). Unlike regular inverse problems, we do not assume knowledge about the measurement function (i.e., the mapping from real images to user-created guides in RBG pixels is unknown), so techniques for solving inverse problems with score-based models (Dhariwal & Nichol, 2021; Kawar et al., 2021) and methods requiring paired datasets (Isola et al., 2017; Zhu et al., 2017) do not apply here.
Procedure. Our method, SDEdit, uses the fact that the reverse SDE can be solved not only from to = 1, but also from any intermediate time to E (0, - an approach not studied by previous SDE-based generative models. We need to find proper initialization from our guides from which we can solve the reverse SDE to obtain desirable, realistic, and faithful images. For any given guide x (9), we define the SDEdit procedure as follows:
Sample ( (to) ~ (xxl)) o (to) then produce x(0) by iterating Equation 4.
We use SDEdit(xx (x((); to. f) to denote the above procedure. Essentially, SDEdit selects particular time to. add Gaussian noise of standard deviation (²(()) to the guide x(s) and then solves the corresponding reverse SDE at 0 to produce the synthesized x())
Apart from the discretization steps taken by the SDE solver, the key hyperparameter for SDEdit is lu- the time from which we begin the image synthesis procedure in the reverse SDE. In the following, we describe
trade-off that allows us to select reasonable values of to-
Realism-fal faithfulnesss trade-off. We note that for properly trained SDE models, there is a realism- faithfulness trade-off when choosing different values of to. To illustrate this, we focus on the LSUN dataset, and use high-level stroke paintings as guides to perform stroke-based image generation. We provide experimental details in Appendix D.2. We consider different choices of to E [0, 1] for the same input. To quantify realism, we adopt neural methods for comparing image distributions, such as the Kernel Inception Score (KID; Binkowski et al., 2018). If the KID between synthesized images and real images are low, then the synthesized images are realistic. For faithfulness, we measure the squared L2 distance between the synthesized images and the guides (()) From Fig. 3, we observe increased realism but decreased faithfulness as to increases.
The
trade-off can be interpreted from another angle. If the guide is far from any realistic images, then we must tolerate least a certain level of deviation from the guide (non-
4

---

Published as a conference paper at ICLR 2022
Input (guide)
StyleGANZ-ADA
In-domain
In-domain
c4c
SDEdit
User eceated
Algorithm
sinmllated
Figure 4: SDEdit generates more realistic and faithful images than state-of-the-ard GAN-based mod- els stroke-based generation (LSUN bedroom). The guide in the first two rows created by human and the ones in the last two rows are simulated by algorithm.
faithfulness) in order to produce realistic image. This is illustrated in the following proposition.
Proposition 1. Assume that [|, t)ll2 S for all x E X and t E [0, 11. Then for all å E 1) with probability at least (1-6).
+ +d x((9) SDEdit(xx()) to || o² (4) log 2og (5)
where d is the number of dimensions of x(9).
We provide the proof in Appendix A. On a high-level, the difference from the guides and the syn- thesized images can be decomposed into the outputs the score and random Gaussian noise; both would increase as to increases, and thus the difference becomes greater. The above proposition sug- gests that for the image to be realistic with high probability, we must have large enough to. On the flip side, if to is too large, then the faithfulness the guide deteriorates, and SDEdit will produce random realistic images (with the extreme case being unconditional image synthesis).
Choice of fg. We note that the quality of the guide may affect the overall quality of the synthesized image. For reasonable guides, we find that to E [0.3, 0.6] works well. However, if the guide is an image with only white pixels, then even the closest "realistic" samples from the model distribution can be quite far, and we must sacrifice faithfulness for better realism by choosing a large to. In interactive settings (where user draws a sketch-based guide), we can initialize lo [0.3, 0.6). syn- thesize candidate with SDEdit, and ask the user whether the sample should be more faithful or more realistic; from the responses, we can obtain a reasonable to via binary search. In large-scale non-interactive settings (where binary search on a randomly selected image to obtain to and subsequently fix tn for all guides in the same task. Although different guides could potentially have different optimal to, we empirically observe that the shared to works well for all reasonable guides in the same task.
given set of produced guides), can perform similar
Detailed algorithm and extensions. We present the general algorithm for VE-SDE in Algo- rithm 1. Due space limit, describe detailed algorithm for VP-SDE in Appendix C. Es- sentially, the algorithm is Euler-Maruyana method for solving SDEdit(x (o, 0). For cases where we wish to keep certain parts of the synthesized images to be identical to that of the guides, can also introduce additional channel that masks out parts of the image do not wish to edit. This is slight modification to the SDEdit procedure mentioned in the main text, and we discuss the details in Appendix C.2.

---

Published as a conference paper at ICLR 2022
Algorithm 1 Guided image synthesis and editing with SDEdit (VE-SDE)
Require: x(9) (guide), to (SDE papeer-prameter.... N (total denoising steps)
At <- t
2 ~ (0, I)
- [(o)z
for n + N do
t - 1
z ~ (0
6 < V (t) - oz (t- At)
x + x + é (x, +
end for Return x
RELATED WORK
Conditional GANs. Conditional GANs for image editing (Isola et al., 2017; Zhu et al., 2017; Jo & Park, 2019; Liu et al., 2021) learn to directly generate an image based on a user input, and have demonstrated success on variety of tasks including image synthesis and editing (Portenier et al., 2018; Chen & Koltun, 2017; Dekel et al., 2018; Wang et al., 2018; Park et al., 2019; Zhu et al., 2020b; Jo & Park, 2019; Liu et al., 2021), inpainting (Pathak al., 2016; lizuka et al., 2017: Yang et al., 2017; Liu et al., 2018). photo colorization (Zhang et al., 2016; Larsson et al., 2016; Zhang al., 2017; He al., 2018), semantic image texture and geometry synthesis (Zhou et al., 2018; Guérin et al., 2017; Xian et al., 2018). They have also achieved strong performance on image edit- ing using user sketch or color (Jo & Park, 2019; Liu al., 2021; Sangkloy al., 2017). However, conditional models have to be trained on both original and edited images, thus requiring data collec- tion and model re- re-training for new editing tasks. Thus, applying such methods to on-the-fly image manipulation is still challenging since a new model needs to be trained for each new application. Unlike conditional GANs, SDEdit only requires training on the original image. As such, it can be directly applied to various editing tasks at test time as illustrated in Fig. 1.
GANs inversion and editing. Another mainstream approach to image editing involves GAN in- version (Zhu et al., 2016; Brock et al., 2017), where the input is first projected into the latent space of an unconditional GAN before synthesizing a new image from the modified latent code. Sev- eral methods have been proposed in this direction, including fine-tuning network weights for each image (Bau et al., 2019a; Pan et al., 2020; Roich et al., 2021), choosing better or multiple layers to project and edit (Abdal et al., 2019; 2020; Gu et al., 2020; Wu et al., 2021), designing better encoders (Richardson et al., 2021; Tov et al., 2021), 1), modeling image coruptiion and transforma- tions (Anirudh et al., 2020; Huh et al., 2020), and discovering meaningful latent directions (Shen et al., 2020; Goetschalckx et al., 2019; Jahanian et al., 2020; Härkönen et al., 2020). However, these methods need to define different loss functions for different tasks. They also require GAN inversion, which can be inefficient and inaccurate for various datasets (Huh et al., 2020; Karras et al., 2020b; Bau al., 2019b; Xu et al., 2021).
Other generative models. Recent advances in training non-normalized probabilistic models, such as score-based generative models (Song & Ermon, 2019; 2020; Song et al., 2021; Ho et al., 2020; Song et al., 2020; Jolicoeur Martineu et al., 2021) and energy-based models (Ackley et al., 1985; Gao et al., 2017: Du & Mordatch, 2019; Xie et al., 2018; 2016; Song & Kingma, 2021), have achieved comparable image sample quality as GANs. However, most of the prior works in this direction have focused on unconditional image generation and density estimation, and state-of-the- art techniques for image editing and synthesis are still dominated by GAN-based methods. In this work, we focus on the recently emerged generative modeling with stochastic differential equations (SDE), and study its application to controllable image editing and synthesis tasks. A concurrent work (Choi et al., 2021) performs conditional image synthesis with diffusion models, where the conditions can be represented as the known function of the underlying true image.

---

Published as a conference paper at ICLR 2022
Inpet (gud))
Generated
LSUN bedroom
LSUN church
ColobA
Figure 5: SDEdit can generate realistic, faithful and diverse images for given stroke input drawn by human.
EXPERIMENTS
In this section, we show that SDEdit is able outperform state-of-the-ar GAN-based models on stroke-based image synthesis and editing well as image compositing. Both SDEdit and the base- lines use publicly available pre-trained checkpoints. Based on the availability of open-sourced SDE checkpoints, use VP-SDE for experiments LSUN datasets, and VE-SDE for experiments CelebA-HQ.
Baselines
Faithfulocss (L2)
SDEdit realistic (MTuk)
SDEdit satisfactory (Mturk)
In-domain GAN-]
101.18
94. 15,9%%%%
89.48%
In-domain GAN-2
57.11
97.87%
89.51
StylGGAN-AAA
68. 12
98.09 09.09%%
91.72%
ede
53.76
80.34%
75.43%
SDEdit
32.55
Table 1: SDEdit outperforms all the GAN baselines on stroke-based generation on LSUN (bed- room). The input strokes are created by human users. The rightmost two columns stand for the percentage of MTurk workers that prefer SDEdit the baseline for pairwise comparison.
Evaluation metrics. We evaluate the editing results based on realism and faäthfulness. quan- tify realism, use Kernel Inception Score (KID) between the generated images and the target realistic image dataset (details Appendix D.2), and pairwise human evaluation between different approaches with Amazon Mechanical Turk (MTurkk). To quantify faithfulnesss, report the L2 distance summed over all pixels between the guide and the edited output image normalized to [0,1]. We also consider LPIPS (Zhang et al., 2018) and MTurk human evaluation for certain experiments. To quantify the overall human satisfaction score (realism + faithfulness)) we leverage MTTurk human evaluation perform pairwise comparsion between the baselines and SDEdit (see Appendix F).
5.1 STROKE- STROKE-BASED IMAGE SYNTHESIS
Given input stroke painting, goal is to generate realistic and faithful image when paired data is available. We consider stroke painting guides created by human users (see Fig. 5). At same time, also propose algorithm to automatically simulate stroke paintings based on a source image (see Fig. 4), allowing us to perform large scale quantitative evaluations for SDEdit. We provide more details in Appendix D.2.
Baselines.
For comparison, we choose three state-of of fee-the-a art GAN-based image editing and synthe- sis methods our baselines. Our first baseline the image projection method used in StyleGAN2- ADA' (Karras et al., 2020a), where inversion is done in the W space of StyleGANs by minimizing the perceptual loss. Our second baseline is in-domain GAN (Zhu et al., 2020a), where inversion is accomplished by running optimization steps on top of an encoder. Specifically, we consider two versions the in-domain GAN inversion techniques: the first one (denoted as In-domain GAN-1)
httpesss/githhb..cmm///bbbb/bbb Phttps: thub. htttsa///ggttto...oo////erreeeee

---

Published as a conference paper at ICLR 2022
Source
Ingut (guide)
GAN bosslinne
SDEai (Ours)
Figure 6: Stroke-based image editing with SDEdit on LSUN bedroom, CelebA-HQ, and LSUN church datasets. For comparison, show the results of GAN baselines, where results for LSUN bedroom and CelebA-HQ are obtained by in-domain GAN (the leftmost panels), and results for LSUN church are from StyleGAN2-ADA (the rightmost 3 panels). We observe that SDEdit is able to produce more faithful and realistic editing compared to the baselines.
only uses the encoder to maximize the inversion speed, whereas the second (denoted as In-domain GAN-2) runs additional optimization steps to maximize the inversion accuracy. Our third baseline e4e² (Tov et al., 2021), whose encoder objective is explicitly designed to balance between perceptual quality and editability by encouraging to invert images close to space a pretrained StyleGAN model.
Results. We present qualitative comparison results Fig. 4. We observe that all base- lines struggle generate realistic images based on stroke painting inputs whereas SDEdit suc- cessfully generates realistic images that pre- serve semantics of the input stroke painting, As shown in Fig. 5, SDEdit can also syn- thesize diverse images for the same input. We present quantitative comparison results us- ing user-created stroke guides in Table and algorithm-simulatee simulated stroke guides in Table 2. We report the L2 distance for faithfulness com- parison, and leverage MTurr (see Appendix F) or KID scores for realism comparison. To quan- tify the overall human satisfaction score (faith- fulness realism), MTuurk workers to perform another 3000 pair- wise comparisons between the baselines and SDEdit based on both faithfulness and realism. We observe that SDEdit outperforms GAN baselines on all the evaluation metrics, beating the basclines by more than 80 % on realism scores and 75% on overall satisfaction scores. We provide more experimental details in Appendix C and more results in Appendix E.
LSUN Bedroom
LSUN Church
Methods
L2 J
KID
L2 I KID.
In-domain GAN-1
105.23
0.1147
In-domain GAN-2
76.11
0,2070
StyleGAN2-ADA
74.03
0.1750
72.41
0,1544
e4e SDEdit (ours)
52.40
0.0464
68.53
0,0354
36.76
0.0030
37.67
0.0156
Table 2: SDEdit outperforms all the GAN base- lines on both faithfulness and realism for stroke- based image generation. The input strokes are generated with the stroke-sinulation algorithm. KID is computed using the generated images and the corresponding validation sets (see Ap- pendix D.2).
ask different set of
5.2 FLEXIBLE IMAGE EDITING
In this section, show that SDEdit is able outperform existing GAN- models image editing tasks. We focus on LSUN (bedroom, church) and CelebA-HQ datasets, and provide more details the experimental setup in the Appendix D.
Stroke-based image editing. Given image with stroke edits, we want generate realistic and faithful image based on the user edit. We consider the same GAN-based baselines (Zhu et al., 2020a; Karras al., 2020a; Tov et al., 2021) as our previous experiment. As shown in Fig. 6,
8

---

Published as a conference paper at ICLR 2022
Poisson Dlending
Laplacian Blending
Imput (guide)
SyHeGANZAAAA
SDEak
Traditional Blending
GIAN brsclines
Figure 7: SDEdit is able to achieve realistic while more faithful editing results compared to tradi- tional blending and recent GAN-based approaches for image compositing on CelehA-HQ. Quanti- tative results reported in Table 3.
La +
SDEdit more realistic
SDEdit more satisfactory (Mturk) f
LPIPS (masked)
Methods
(faithfulness)
(Mturk) f
Luplacian Blending
68.45
75.27%
83.73%
0.09
Poisson Blending
63.04
75.60%
82.18%
0.05
In-domain GAN
36.67
53.08%
73.53%
0.23
StyleGAN2-ADA
69.38
74.12%
83.43%
0.21
<4c SDEdit (ours)
53.90
43.67%
66.00%
0.33
21.70
0.03
Table 3: Image compositing experiments on CelebA-HQ. The middle two columns indicate the per- centage of MTTurk workers that prefer SDEdit. We also report the masked L.PISS distance between edited and unchanged images to quantify undesired changes outside the masks. We observe that SDEdit is able achieve realistic editing while being more faithful than the baselines, beating the baseline by up to 83.73% % on overall satisfaction score by human evaluators.
results generated by the baselines tend to introduce undesired modifications, occasionally making the region outside the stroke blurry. contrast, SDEdit generates image edits that are both realistic and faithful to the input, while avoiding making undesired modifications. We provide extra results in Appendix
Image compositing. We focus compositing images the CelebA-HQ dataset (Karras al., 2017). Given an image randomly sampled from the dataset, we ask users to specify how they want the edited image to look like using pixel patches copied from other reference images well as the pixels they want to perform modifications (see Fig. 7). We compare our method with traditional blending algorithms (Burt Adelson, 1987; Pérez al., 2003) and the same GAN baselines con- sidered previously. We perform qualitative comparison in Fig. 7. For quantitative comparison, we report the Lg distance to quantify faithfulness. To quantify realism, we ask MTurk workers to per- form 1500 pairwise comparisons between the baselines and SDEdit. To quantify user satisfaction score (faithfulness + realism), we ask different workers to perform another 1500 pairwise compar- isons against SDEdit. To quantify undesired changes (e.g. change of identity), we follow Bau et al. (2020) to compute masked LPIPS (Zhang et al., 2018). As evidenced in Table 3, we observe that SDEdit is able to generate both faithful and realistic images with much better LPIPS scores than the baselines, outperforming the baselines by up to 83.73% % on overall satisfaction score and 75.60 % on realism. Although realism score is marginally lower than e4e, images generated by SDEdit are more faithful and more satisfying overall. We present more experiment details in Appendix D.
6 CONCLUSION
We propose Stochastic Differential Editing (SDEdit), a guided image editing and synthesis method via generative modeling of images with stochastic differential equations (SDEs) allowing for bal- anced realism and faithfulness. Unlike image editing techniques via GAN inversion, our method does not require task-specific optimization algorithms for reconstructing inputs, and particularly suitable for datasets or tasks where GAN inversion losses are hard to design optimize. Unlike conditional GANs, our method does not require collecting new datasets for the "guide" images or re-training models, both of which could be expensive or time-consuming We demonstrate that SDEdit outperforms existing GAN-based methods on variety of image synthesis and editing tasks.

---

Published as a conference paper at ICLR 2022
Acknowledgments. The authors would like to thank Kristy Choi for proofreading. This search was in part supported by NSF (#1651565, #1522054, #1 #1733686), ONR (N00014-19-1-2145, AFOSR (FA9550-19-1 11750001----0244 1-0024,, ARO (W91INF-15-11) 15-1-4779,, Autodesk, Google, Bosch, Stanford In- stitute for Human-Centered AT (HAT), Stanford Center for Integrated Facility Engineering (CIFE), Amazon Research Award (ARA), and Amazon AWS. Yang Song is supported by the Apple Phl) Fellowship in AI/ML. J.-Y. Zhu is partly supported by Naver Corporation.
ETHICS STATEMENT
In this work, we propose SDEdit, which is a new image synthesis and editing methods based on generative stochastic differential equations (SDEs). our experiments, all the considered datasets are open-soureed and publicly available, being used under permission. Similar to commonly seen deep-learning based image synthesis and editing algorithms, our method has both positive and nega- tive societal impacts depending on the applications and usages. On the positive side, SDEdit enables everyday users with or without artistic expertise create and edit photo-realistic images with min- imum effort, lowering the barrier to entry for visual content creation. On the other hand, SDEdit can be used to generate high-quality edited images that are hard to be distinguished from real ones by humans, which could be used in malicious ways to deceive humans and spread misinformation. Similar to commonly seen deep-learning models (such as GAN-based methods for face-editing), SDEdit might be exploited by malicious users with potential negative impacts. In our code release, we will explicitly specify allowable uses of our system with appropriate licenses.
We also notice that forensic methods for detecting fake machine-generated images mostly focus distinguishing samples generated by GAN-based approaches. Due to the different underlying nature between GANs and generative SDEs, observe that state-of-the-ar approaches for detecting fake images generated by GANs (Wang et al., 2020) struggle to distinguish fake samples generated by SDE-based models. For instance, on the LSUN bedroom dataset, it only successfully detects less than 3% of SDEdit-generatee images whereas being able to distinguish up to 93% on GAN-based generation. Based on these observations, we believe developing forensic methods for SDE-based models is also critical as SDE-based methods become more prevalent.
For human evaluation experiments, we leveraged Amazon Mechanical Turk (MTurkk). For each worker, the evaluation HIT contains 15 pairwise comparison questions for comparing edited images. The reward per task is kept as 0.2$. Since each task takes around minute, the wage is around 12$ per hour. We provide more details on Human evaluation experiments in Appendix F. We also note that the bias of human evaluators (MTurk workers) and the bias of users (through the input "guidance") could potentially affect the evaluation metries and results used to track the progress towards guided image synthesis and editing.
REPRODUCIBILITY STATEMENT
1. Our code is released at https: /tt:://gtttttt com////mongruuu//Seeaaeo.
2. We open source datasets and SDE checkpoints the corresponding datasets. We did not train any SDE models.
3. Proofs are provided in Appendix A.
4. Extra details on SDEdit and pseudocode are provided in Appendix C.
5. Details experimental settings are provided in Appendix D.
6. Extra experimental results are provided in Appendix E.
7. Details on human evaluation are provided in Appendix F.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan+t: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296-83960 2020.
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147- 1985.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12 (3):113-326, 1982.
Rushil Anirudh, Jayaraman J Thiagarajan, Bhavya Kailkhura, and Peer-Timo Bremer. Mimicgan: Robust projection onto image manifolds with corruption mimicking. International Journal of Computer Vision, pp. 1-19, 2020.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM SIGGRAPH, 38(4):-111, 2019a.
David Bau, Jun- Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what a gan cannot generate. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4502 4511, 2019b.
David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba. Rewriting deep generative model. In European Conference on Computer Vision (ECCV), 2020.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 1001.0100101111 01101, 2018.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In International Conference on Learning Representations (ICLR), 2017.
Peter J Burt and Edward H Adelson. The laplacian pyramid as compact image code. In Readings in computer vision, pp. 671 671-679. Elsevier. 1987.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. lvvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108. 029388 2021.
Tali Dekel, Chuang Gan, Dilip Krishnan, Ce Liu, and William T Freeman. Sparse, smart contours represent and edit images. IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. vrXiv:2105.0523: 2021.
arXiv preprint
Yilun and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:/903. (eX.0:9903.08689 2019.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Learning energy-based models as generative convnets via multi-grid modeling and sampling. arXiv e-prints, pp. arXiv-1709, 2017.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions of cognitive image properties. In IEEE International Conference on Computer Vision (ICCV), 2019.
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Éric Guérin, Julie Digne, Éric Galin, Adrien Peytavie, Christian Wolf, Bedrich Benes, and Bemoundefinedt Martinez. Interactive example-based terrain authoring with conditional generative adversarial networks. ACM Transactions Graphics (TOG), 36(6), 2017.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In Advances in Neural Information Processing Systems, 2020.
11

---

Published as a conference paper at ICLR 2022
Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. Deep exemplar-based colorization. ACM Transactions on Graphics (TOG), 37,4);1-16, 2018.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabílistic models. arXiv:2006. / 1239, 2020.
arXiv preprint
Minyoung Huh, Richard Zhang. Jun-Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and pro- jecting images into class-conditional generative networks. In European Conference on Computer Vision (ECCV), 2020.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions Graphics (TOG), 36(4):107, 2017.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-iman translation with conditional adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Ali Jahamian, Lucy Chai, and Phillip Isola. On tte"stcerability" of generative adversarial networks. In Interna- tional Conference on Learning Representattions (ICI.), 2020.
Youngjoo Jo and Jongyoul Park. Sc-fegan: Face editing generative adversarial network with user's sketch and color. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1755,1753, 2019.
Alexia Jolicoeur-Mann Martineau, Rémi Piché- Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. Adversar- ial score matching and improved sampling for image generation. In International Conference on Learning Representations, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 17.0v 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint uxXiv:2006.0661 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020b.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. arXiv preprint 1455
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic coloriza- tion. In European Conference Computer Vision (ECCV), 2016.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection, Annals of Statistics, pp. 1302-1338, 2000.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpaint- ing for irregular holes using partial convolutions. In Eurupean Conference on Computer Vision (ECCV), 2018.
Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao, Bin Jiang, and Wei Liu. Defloc- net: Deep image editing via flexible low-level controls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10765-10774, 2021.
Xingang Pan, Xiaohang Zhan, Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In European Conference on Computer Vision, 2020.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In ACM SIGGRAPH, pp. 313-318, 2003.
12

---

Published as a conference paper at ICLR 2022
Tiziano Portenier, Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker. Faceshop: Deep sketch-based face image editing. ACM Transactions on Graphics (TOG), 37(4), 2018.
Flad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. arXiv preprint arxiv:2 2021.
Patsor Sangkloy. Jingwan Lu, Chen Fang. Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jascha Sohl- Dickkseiin Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 1.503035855 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint
arXiv: 200.:20010002502, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurlPS), 2019.
Yang Song and Stefano Ermon. Improved techniques training score-base based generative models. arXiv preprint arXiv:2006.000 09011, 2020.
Yang Song and Diederik P Kingma. How train your energy-based models. arXiv preprint arXiv:2101.0822 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basee generative modeling through stochastic differential equations. International Conference on Learning Representations (ICLR), 2021.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphicx (TOG), 40(4):1-14, 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661-167,, 2011.
Sheng-Ye Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are surprisingly easy spot...for now. In CVPR, 2020.
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Zongze Wu, Dani Lischinski. and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2021.
Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj. Jingwan La, Chen Fang. Fisher Yu, and James Hays. Texturegan: Controlling deep image synthesis with texture patches. In IEFF Conference on Computer Vision and Patiern Recognition (CVPR), 2018.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingmian Wu. A theory of generative convnct. In International Conference on Machine Learning, pp. 2635-2644, PMLR, 2016.
Jianwen Xie. Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolci Zhou. Generative hierarchical features from synthesizing images. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recog- nition, pp. 4432 4442, 2021.
Chao Yang. Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
13

---

Published as a conference paper at ICLR 2022
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference Computer Vision (ECCV), 2016.
Richard Zhang. Jun- Yan Zhu, Phillip Isola, Xinyang Geng. Angela Lin, Tianhe Yu, and Alexei A Efros. Real-time user-guided image colorization with learned deep priors. ACM Transactions on Graphics (TOG), 9()), 2017.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or. Hui Huang. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics (TOG), 37(4), 2018.
Jiapeng Zhu, Yujum Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European Conference on Computer Vision (ECCV), 2020a.
Jun- Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on natural image manifold. In European Conference on Computer Vision (ECCV), 2016.
Jun- Yan Zhu, Taesung Park, Phillip Isola, Alexei Efros. Unpaired sangetto-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaptive normalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020b.
14

---

Published as a conference paper at ICLR 2022
A PROOFS
Proposition Assume that [|80(x, t)ll2 I for all x E & and t E [0, 1]. Then for all 8 E (0,1) with probability at least (1-6),
x SDEdit(xx 1o. o1 5 (C)) 2/ dd log -2 log 6)
(5)
where d is the number of dimensions of
Proof. Denote x(9) x())(0) (SEEdit(x())) 0). then
1 | (x (t)
(t)) x00
dt
(6)
dt
// d[(2(()]
de + v /dlo2(1)
8o (0 1;0)
dw
dt
dt
so (x, (x) / d(o2(())
l0 /d(a2(1)
dt
dw
(8)
dt
dt
From the assumption over sx 0), the first term is not greater than
c clf.1 dlo c
dt at Co (oo(
dt
where equality could only happen when each score output has a squared L2 norm of and they are linearly dependent to other. The second term is independent to the first term it only concerns random noise; this is equal to the squared L2 norm of a random variable from Wiener process time f = O, with marginal distribution being o (this marginal does not depend on the discretization steps in Euler-Maruyama), The squared L2 norm of e divided by o²((n) is di-distribution with d-degrees of freedom. From Laurent & Massart (2000), Lemma we have the following one-sided tail bound:
Pr[Iee] /02 2 d 2 log log S exp(log d) 8.
Therefore, with probability least (1 8), we have that:
(9) (1)) - -x) < (1))(())))) + 2/ log ā - log 8),
(10)
completing the proof.
B EXTRA ABLATION STUDIES
In this section, we perform extra ablation studies and analysis for SDEdit.
B. ANALYSIS ON THE QUALITY OF USER GUIDE
As discussed in Section 3, if the guide far from any realistic images (e.g., random noise or has an unreasonable composition). then we must tolerate at least a certain level of deviation from the guide (non-faithfulness) order produce realistic image.
For practical applications, we perform extra ablation studies on how the quality of guided stroke would affect the results in Fig. 8, Fig. and Table Specifically, in Fig. we consider stroke input of I) human face with limited detail for CelebA-HQ model, 2) a human face with spikes for a CelebA-HQ model, 3) a building with limited detail for a LSUN-church model, 4) a horse for LSUN-church model. We observe that SDEdit is in general tolerant to different kinds of user inputs. Table 4, we quantitatively analyze the effect of user guide quality using simulated algorithm uses
stroke paintings input. Described in Appendix D.2, the samaa--trook-simalne
15

---

Published as a conference paper at ICLR 2022
different numbers of colors generate stroke guides with different levels of detail. We compare SDEdit with baselines qualitatively in Fig. 9 and quantitatively in Table 4. Similarly, we observe that SDEdit has high tolerance to input guides and consistently outperforms the baselines across all setups this experiment.
Inpul (guide)
Output
CelebA-HI
LSUN-Chureh
Figure 8: Analysis the quality of guide for stoke- based image synthesis. We observe that SDEdit is in general tolerant to different kinds of user inputs.
# of colors =
of colors
6
# of colors 16 # of colors 30
# of colors 50
Siumllated stroe
SyyAGAN2 ADA
SDEdi
(Ouns)
Figure 9: Analysis on the quality of user guide for stoke-based image synthesis. We observe that SDEdit is in general tolerant to different kinds of user inputs.
16

---

Published as a conference paper at ICLR 2022
StyleGAN2-ADA
e4e
SDEdit (Ours)
# of colors
KID 4
L2 4
KID 4
L2
KID
L2 4
0.1588
67.22
0.0379
70.73
0.0233
36.00
0.1544
72.41
0,0354
68.53
0.0156
37.67
16
0.0923
69.52
0,0319
68.20
0.0135
37.70
30
0.0911
67.11
0.0304
68.66
0.0128
37.42
50
0.0922
65.28
0.0307
68.80
0.0126
37.40
Table 4: We compare SDEdit with baselines quantitatively on LSUN-church dataset on stroke- based generation. "# of colors" denotes the number of colors used to generate the synthetic stroke paintings, with fewer colors corresponding to a less accurate and less detailed input guide (see Fig. 9). We observe that SDEdit consistently achieves more realistic and more faithful outputs and outperforms the baselines across all setups.
B.2 FLEXIBLE IMAGE EDITING WITH SDEDIT
In this section, we perform extra image editing experiments including editing closing eyes Fig. 10, opening mouth, and changing lip color Fig. 11. We observe that SDEdit can still achieve reasonable editing results, which shows that SDEdit is capable of flexible image editing tasks.
Source
Input (guide)
Output
Figure 10: Flexible image editing closing eyes with SDEdit.
Changing lip color
Opening mouth
Smile with teeth
Input (guide)
Input (guide)
Input (guide)
Source
Output
Output
Oulput
Figure 11: Flexible image editing on mouth with SDEdit.
B.3 ANALYSIS ON to
In this section, provide extra analysis the effect of to (see Fig. 12). As illustrated in Fig. 3. we can tune to to tradeoff between faithfulness and realism wethism-with with smaller to corresponding to more faithful but less realistic generated image. If we want to keep the brown stroke in Fig. 12, can reduce to to increase its faithfulness which could potentially decrease its realism. Additional analysis can be found in Appendix D.2.
17

---

Published as a conference paper at ICLR 2022
b=00
1==0.1
1=02
k=0,3
1,=0.35
4=0.1
Input (guide)
ly-00.45
1,m0.5
1gm0.7
1gm018
lym0,9
Figure 12: Extra analysis on lu- As ty increases, the generated images become more realistic while less faithful.
B.4 EXTRA COMPARISON WITH OTHER BASELINES
We perform extra comparison with SC-FEGAN (Jo & Park, 2019) in Fig. 13. We observe that
SDEdit is able to have more realistic results than SC-FEGAN (Jo & Park, 2019) when using the same stroke input guide. We also present results for SC-FEGAN (Jo & Park, 2019) where extra sketch together with stroke as the input guide (see Fig. 14). We observe that SDEdit still able to outperform SC-FEGAN in terms of realism even when SC-FEGAN is using both sketch and stroke as the input guide.
Input (guide)
SC-FEGAN
SDDEdit (ouns)
Imput (guide)
SC-FEGAN
SDEdit (ours)
Figure 13: Comparison with SC-FEGAN (Jo & Park, 2019) on stroke-based image synthesis and editing. We observe that SDEdit is able to generate realistic results than SC-FEGAN.
Stetch
Stoke
Ouput
Inpas (gsice)
Figure 14: Stroke-based editing for SC-FEGAN (Jo & Park, 2019) using both stroke and extra sketch the input guide. We observe that SDEdit still outperforms SC-FEGAN using only stroke as the input guide.
B.5 COMPARISON WITH SONG ET AL. (2021)
Methods proposed by Song et al. (2021) introduce an extra noise-conditioned classifier for condi- tional generation and the performance of the classifier is critical the conditional generation perfor- mance. Their settings are more similar to regular inverse problems where the measurement function known, which is discussed in Section 3. Since do not have known "measurement" function
18

---

Published as a conference paper at ICLR 2022
for user-generated guides, their approach cannot be directly applied user user-guided image synthe- sis or editing in the form manipulating pixel RGB values. To deal with this limitation, SDEdit input and modifics tn accordingly- approach different from Song et al. (2021) (which always have the same initialization). This technique allows SDEdit achieve faithful and realistic image editing generation results without extra task-specific model learning (e.g., an additional classifier in Song et al. (2021)).
initializes the reverse SDE based
For practical applications, compare with Song al. (2021) on stroke-based image synthesis and editing where we do not learn an extra noise-conditionel classifier (see Fig. 15). In fact, we are also unable learn the condiicondi classifier since do have a known "measurement" function for user-g generated guides and we only have one random user input guide instead of a dataset of input guide. We observe that this application of Song al. (2021) fails to generate faithful results by performing random inpainting (see Fig. 15). SDEdit, on the other hand, generates both realistic and faithful images without learning extra task task-specific models (e.g., an additional classifier) and can be directly applied to pretrained SDE-based generative models, allowing for guided image synthesis and editing using SDE-based models. believe this shows the novelty and contribution of SDEdit.
Inpur (guide)
(Song al)
SDFdd (ours)
Input (guide)
(Song et. al.)
SDEddi (ours)
Comparison with Song et (2021) on troke-based image synthesis and editing. We observe that SDEdit is able to generate more faithful results than Song al. (2021) without training an extra task-specific model (e.g., -,, an additional classifier).
Figure 15:
C DETAILS ON SDEDIT
C.I DETAILS ON THE VP AND VE SDES
We follow the definitions of VE and VP SDEs in Song et al. (2021), and adopt the same settings therein.
VE-SDE In particular, for the VE SDE, we choose
1
(t))
min
8> 0
where miin 0.01 and Faax 380, 378, 348, 1348 for LSUN churches, bedroom, FFHO/CelebA-- HQ 256 x 256, and FFHQ 1024 x 1024 datasets respectively.
VP-SDE For the VP SDE, it takes the form of dx(t) - (())x(t))t + /B(t)dw(tt)
(11)
where ((t) is positive function. In experiments, we follow Song et al. (2021); Ho et al. (2020);
Dhariwal Nichol (2021) and set
((t) Amin + 1(3maxx Buin),
For SDE trained by Song et (2021); Ho al. (2020) use 3mia 0.1 and 3 20; for SDE trained by Dhariwal & Nichol (2021), the model learns to rescale the variance based on the same choices of Amin and Bmax- We always have pı (x) R A/0,, I) under these settings.
Solving the reverse VP SDE is similar to solving the reverse VE SDE. Specifically, we follow the iteration rule below:
1
X
Xx. + Ats
+ ((tt)) Zn.
(12)
VI I ((tn)at
where XN NN(0,, I), 2n I) and n N, N 1....,
19

---

Published as a conference paper at ICLR 2022
C.2 DETAILS ON STOCHASTIC DIFFERENTIAL EDITING
In generation the process detailed in Algorithm can also be repeated for K number of times as detailed in Algorithm 2. Note that Algorithm 1 is special recover Algorithm I. For VE-SDE, Algorithm 2 converts a stroke painting to a photo-realistic image, which typically modifies all pixels of the input. However, in cases such as image compositing and stroke-based editing, certain regions of the input are already photo-realistic and therefore we hope to leave these regions intact. To represent a specific region, we use binary mask N {0, 1] x x that evaluates to for editable pixels and O otherwise. We can generalize Algorithm 2 to restrict editing the region defined by 52
of Algorithm 2: when K =
For editable regions, we perturb the input image with the forward SDE and generate edits by re- versing the SDE, using the same procedure in Algorithm usual but design the reverse procedure carefully so that it is guaranteed recover the input. Specif- ically, suppose x E CC*H* xW is an input image of height H, width W, and with C' channels. Our algorithm first perturbs x(0) Afterwards, we denoise x(tu) with separate methods for @ x(t) and (1 St) o x(t). where O denotes the element wise product and 0 s t to- For Ω (t)), we simulate the reverse SDE (Song et al., 2021) y and project the results by element-wis wise multiplication with Ω. For (1 - 22) @ x(). we set it (1 .2) @ (x + (l)z), where z No0, I). Here we gradually reduce the noise magnitude according to (t) to make sure Ω @ x(t) and (1 52) @ x(t) have comparable amount of noise. Moreover, since (t) 0 as
For uneditable regions, we perturb it as
with an SDE running from t 0 till t to to obtain x(to)-
0, this ensures that (1 S2) x(t) converges to (1 S2) x,
keeping the uneditable part of x intact. The complete SDEdit method (for VE-SDEs) is given in Algorithm 3. We provide algorithm for VP-SDEs in Algorithm and the corresponding masked version in Algorithm 5.
With different inputs to Algorithm or Algorithm 5, we can perform multiple image synthesis and editing tasks with single unified approach, including but not limited to the following:
Stroke-based image synthesis: We can recover Algorithm 2 or Algorithm 4 by setting all entries in Z
Stroke-based image editing: Suppose ( is an image marked by strokes, and Ω masks the part that are not stroke pixels. We can reconcile the two parts of x(s) with Algorithm obtain photo-realistic image.
Image compositing: Suppose x(9) is an image superimposed by elements from two im- ages, and Z masks the region that the users do not want to perform editing, can perform image compositing with Algorithm 3 or Algorithm 5.
Algorithm 2 Guided image synthesis and editing (VE-SDE)
Require: x9) (guide), to (SDE hyper-parameter)) N (total denoising steps), K (total repeats)
At < tg N
for 1 to K do z ~ \O,
< x + (to)) for n < N 1
< 10
~ I)
< / (t) at At) At)
x + e² so (x
end for
end for Return x
20

---

Published as a conference paper at ICLR 2022
Algorithm 3 Guided image synthesis and editing with mask (VE-SDE)
Require: x(9) (guide), Ω (mask for edited regions), to (SDE hyper-parameter), N (total denoising
steps), K (total repeats)
Af to N Xo x
for + 1 to K do (0, I]
x f (1 - St) X + 2 +
for N to 1 do
1 4 lo
2 (0, I)
c < v (t) (t - At)
x + (1 - ②2) @ (Xo + 01))) + + + (x+e0ox0 + +2)
end for
end for Return
Algorithm Guided image synthesis and editing (VP-SDE)
Require: x(9) (guide), (SDE hyper-parameter), N (total denoising steps), K (total repeats)
Al < fg
N
(taa) + I==
N
3( nto At)
for k <- 1 to K do
2 .N(
x + Va(to]x + / a(to)z
for N to 1 do
t + to
(O,I))
x VI-R(1)Aa (x (tsgxt))( + /B(t)Atz
end for end for Return X
Algorithm 5 Guided image synthesis and editing with mask (VP-SDE)
Require: x9) (guide), Ω (mask for edited regions), 4 (SDE hyper-parameter), N (total denoising steps), K (total repeats)
At < N
Xg < x
a(tu) +- |I (1- 5( At)
N
for l: <- 1 to K do
2 (N,
x + [11-10 [2) o va(to)xo+ + 16 Valtoxx /I a(to)z)
for n N to do
t to
I)
a(t)
IT (1
x
[1 Q)) (Va(t)xo VI- a(fzz) +Ω 11)) (x (tsgg))s(xx, t)) +
B(t)tt 2)
end for end for Return x
21

---

Published as a conference paper at ICLR 2022
D EXPERIMENTAL SETTINGS
D.1 IMPLEMENTATION DETAILS
Below, we add additional implementation details for each application. We use publicly available pretrained SDE checkpoints provided by Song et al.; Ho et al.; Dhariwal & Nichol. Our code will be publicly available upon publication.
Stroke-basse based image synthesis. this experiment, we use K = I, = 500, to = 0.5, for SDEdit (VP). We find that K to 3 work reasonably well, with larger K generating more realistic images but at a higher computational cost.
For StyleGAN2-ADA, in-domain GAN and e4e, use the official implementation with default parameters to project each input image into the latent space, and subsequently use the obtained latent code produce stroke-based image samples.
Stroke-based image editing. We use K in the experiment for SDEdit (VP). We use to 0.5, N 500 for SDEdit (VP), and lo = 0.45, N = 1000 for SDEdit (VE).
Image compositing. We use CelebA-HO HQ (256x256) (Karras et al., 2017) for image compositing experiments. More specifically, given image from CelebA-HQ, the user will copy pixel patches from other reference images, and also specify the pixels they want to perform modifications, which will be used as the mask in Algorithm 3. In general, the masks are simply the pixels the users have copied pixel patches to. We focus on editing hairstyles and adding glasses. We use an SDEdit model pretrained on FFHQ (Karras al., 2019). We use to 0.35, N 700, K = 1 for SDEdit (VE). We present more results in Appendix E.2.
D.2 SYNTHESIZING STROKE PAINTING
Human-strokesiiuulttoo algorithm We design a buman-stroke-simuulan algorithm in order to perform large scale quantitative analysis stroke-based generation. Given 256 666x256 image, we first apply median filter with kernel size 23 to the image, then reduce the number of colors to using the adaptive palette. We use this algorithm on the validation church outdoor, and subset of randomly selected 6000 images in the CelebA (256 256) test set to produce the stroke painting inputs for Fig. 3a, Table 2 and Table 5. Additionally Fig. 30, Fig. 31 and Fig. 32 show examples of the ground truth images, synthetic stroke paintings, and the corresponding generated images by SDEdit. The simulated stroke paintings resemble the ones drawn by humans and SDEdit is able to generate high quality images based on this synthetic input, while the baselines fail to obtain comparable results.
of LSUN bedroom and LSUN
KID evaluation KID is calculated between the real image from the validation set and the gen- erated images using synthetic stroke paintings (based on the validation set), and the squared Lz distance is calculated between the simulated stroke paintings and the generated images.
Realism-faithfulnee trade-off To search for the sweet spot for realism- faithfulness trade-off as presented in Figure 3a, we select 0.01 and every 0.1 interval from 0.1 to 1 for to and generate images for the LSUN church outdoor dataset. We apply the numan-stroke-simulation algorithm on the original L.SUN church outdoor validation set and generate one stroke painting per image to produce the same input stroke paintings for all choices of to. As shown in Figure 33, this algorithm is sufficient to simulate human stroke painting and we can also observe the trade-off given the same stroke input. KID is calculated between the real image from the validation set and the generated images, and the squared L2 distance is calculated between the simulated stroke paintings and the generated images.
D.3 TRAINING AND INFERENCE TIME
We use open source pretrained SDE models provided by Song et al.; Ho et al.; Dhariwal & Nichol. In general, VP and VE have comparable speeds, and can be slower than encoder-based GAN inversion
22

---

Published as a conference paper at ICLR 2022
methods. For scribble-based generation 256x256 images. SDEdit takes 29.1s generate one image on one 2080Ti GPU. In comparison, StyleGANZ-ADA (Karras et al., 2020a) takes around 72.8s and In-domain GAN 2 (Zhu et al., 2020a) takes 5.2s using the same device and setting. We note that our speed is in general faster than optimization-bn encoder-based GAN inversions. The speed of SDDdiit could be improved by recent works faster SDE sampling.
GAN inversions while slower than
E EXTRA EXPERIMENTAL RESULTS
E.1 EXTRA RESULTS ON LSUN DATASETS
Stroke-based image generation. We present more SDEdit (VP) results on LSUN bedroom in Fig. 21. We use to 0.5, N 500, and realistic images that share the same structure as the input paintings when paired data is provided.
We observe that, SDEdit is able to generate
Stroke-based image editing. We present more SDEdit (VP) results on LSUN bedroom Fig. 22. SDEdit generates image edits that are both realistic and faithful to the user edit, while avoids making undesired modifications on pixels not specified by users. See Appendix D for experimental settings.
E.2 EXTRA RESULTS ON FACE DATASETS
Stroke-based image editing. We provide intermediate step visualizations for SDEdit in Fig. 23. We present extra SDEdit results on CelebA-HQ in Fig. 24. We also presents results on CelebA-HQ (1024x 1024) in Fig. 29. SDEdit generates images that are both realistic and faithful (to the user edit), while avoids introducing undesired modifications on pixels not specified by users. We provide experiment settings Appendix D.
Image compositing, We focus on editing hair styles and adding glasses. We present more SDEdit (VE) results on CelebA-HQ (26x2266) on CelebA-HQ x1024x 1024) in Fig. 28. We observe that SDEdit can generate both faithful and realistic edited images. See Appendix D for experiment settings.
Fig. 25, Fig. 26, and Fig. 27. We also presents results
Attribute classification with stroke-based generation. In order to further evaluate how the mod- els convey user intents with high level user guide, we perform attribute classification on stroke-based generation for human faces. We use the numan--trokk-simmlltee selected 6000 images from CelebA (256x 256) test set to create the stroke inputs, and apply Mi- crosoft Azure Face API to detect tne-ggrined face attributes from the generated images. We choose gender and glasses to conduct binary classification, and hair color to perform multi-class classifica- tion on the images. Images where no face is detected will be counted as providing false and to the classification problems. Table shows the classification accuracy, and SDEdit (VP) outperforms all other basclines in all attributes of choice.
algorithm on subset of randomly
E.3 CLASS-CONDITIONS GENERATION WITH STROKE PAINTING
In addition to user guide, SDEdit is able to also leverage other auxiliary information and models obtain further control of the generation. Following Song et al. (2021) and Dhariwal & Nichol (2021), we present extra experiment on class-conditional generation with SDEdit. Given a time- dependent classifier Pe x), for SDEdit (VE) one can solve the reverse SDE:
dx(()= dlo²(tt) (Vx lx)
d (t)
+ Vx log xi) dt
dw (13)
di
di
and use the same sampling procedure defined in Section 3.
'ttpps: htt/t/gtttbbttt www.//oo///eeaeeeee Samp les /oon/ vi servi cervvices-eee wkstartt-cddde tree/master/ crees/masterrppythhn//ecc.
23

---

Published as a conference paper at ICLR 2022
(a) Dataset image
(b) User guide
(c) GAN output
(d) GAN blending
Figure 16: Post-processing samples from GANs by masking out undesired changes, yet the artifacts are strong at the boundaries even with blending.
Methods
Gender
Glasses
Hair Blond
Hair Black
Hair Grey
In-domain GAN 1
0.5232
0.6355
0.5527
0,5722
0.5398
In-domain GAN 2
0,0202
0.0273
0.1806
0.3158
0,0253
StyleGAN2-ADA
0.0127
0.0153
0.1720
0.3105
0.0145
e4e
0.6175
0.6623
0.6731
0.6510
0.7233
SDEdit (ours)
0.8147
0.9232
0.8487
0.7490
0.8928
Table 5: Attribute classification results with simulated stroke inputs on CelebA. SDEdit (VP) out- performs all baseline methods in all attribute selected in the experiment. Details can be found in Appendix E.2.
For SDEdit (VP), follow the class guidance setting in Dhariwal & Nichol (2021) and solve:
Xn-1 V1 - A( (t)
xx) x Zn
(14)
Fig. shows the ImageNet (256x256) class-conditional generation results using SDEdit (VP). Given the same stroke inputs, SDEdit is capable of generating diverse results that are consistent with the input class labels.
E.4 EXTRA DATASETS
We present additional stroke-based image synthesis results on LSUN cat and horse dataset for SDEdit (VP). Fig. 35 presents the image generation results based on input stroke paintings with various levels of details. faithful to the stroke input on both datasets. Notice that for coarser guide (e.g. the third row in Fig. 35), we choose to slightly sacrifice faithfulness in order to obtain more realistic images by selecting larger to 0.6, while all the other images in Fig. 35 are generated with to 0.5.
can observe that SDEdit produce images that are both realistic and
E.5 EXTRA RESULTS ON BASELINES
SDEdit preserves the un-masked regions automatically, while GANs do not. We tried post- processing samples from GANs by masking out undesired changes, yet the artifacts are strong at the boundaries. We further tried blending GANs (GAN blending) with StyleGAN2-ADA, but the artifacts are still distinguishable (see Fig. 16).
F HUMAN EVALUATION
F.I STROKE-BASED IMAGE GENERATION
Specifically, we synthesize a total of 400 bedroom images from stroke paintings for each method. To quantify sample quality, we ask the workers perform total of 1500 pairwise comparisons against SDEdit to determine which image sample looks more realistic. Each evaluation HIT contains 15 pairwise comparisons against SDEdit, and perform 100 such evaluation tasks. The reward per task kept as 0.2$. Since each task takes around min. the wage is around 12S per hour. For each question, the workers will be shown two images: one generated image from SDEdit and the other
24

---

Published as a conference paper at ICLR 2022
Start!
Figure 17: The instruction shown to MTurk workers for pairwise comparison.
Wichhinage
Lef Irage
Pight image
Figure 18: The UI shown MTurk workers for pairwise comparison.
from the baseline model using the same input. The instruction is: "Which image do you think is more realistic" (see Fig. 17 and Fig. 18).
To quantify user satisfactory score (faithfulness+realism))
ask different set of workers to per- form another 3000 pairwise comparisons against SDEdit. For each question, the workers will be shown three images: the input stroke painting (guide), one generated image from SDEdit based the stroke input, and the other from the baseline model using the same input. Each evaluation HIT contains 15 pairwise comparisons against SDEdit, and we perform 200 such evaluation tasks. The reward per task is kept as 0.2$. Since each task takes around min, the wage is around 12$ per hour. The instruction is: "Given the input painting, how would you imagine this image to look like in reality? Choose the image that looks more reasonable to you. Your selection should based on how realistic and less blurry the image is, and whether shares similarities with the input" (see Fig. 19 and Fig. 20).
About this HIT:
Please only perticipate this HIT it have normal color abeion.
choul take about minuts
You tolo pan on exporimont iwvking Hiuui poroeption You'll coo sorlos of pair of imggoss ocon fi "iee"" genentasd camgae cutar Yiur dhraer which tour painting.
Glven would mouinng reasonble cotration based
took Choore the
similarttie nout.
Start!
Figure 19: The instruction shown MTurk workers for pairwise comparison.
25

---

Published as a conference paper at ICLR 2022
Cren the incut peintin, how reuld this
reasonble medly? image
similrities
Lett image
B ghe image
Figure 20: The UI shown to MTurk workers for pairwise comparison.
F.2 IMAGE COMPOSITING ON CELEBA-HQ
To quantitatively evaluate results, generate 936 images based the user inputs. To quantify realism, we ask MTurk workers to perform 1500 pairwise comparisons against SDEdit pre-trained on FFHQ (Karras al., 2019) to determine which image sample looks more realistic. Each evalu- ation HIT contains 15 pairwise comparisons against SDEdit, and we perform 100 such evaluation tasks. The reward per task is kept 0.2S. Since each task takes around min, the wage around 12S per hour. For each question, the workers will be shown two images: one generated image from SDEdit and the other from the baseline model using the same input. The instruction is: "Which image do you think was more realistic?".
To quantify user satisfactory score (faithfulness realism), we ask different workers to perform another 1500 pairwise comparisons against SDEdit pre-trained FFHQ to decide which generated image matches the content of the inputs more faithfully. Each evaluation HIT contains 15 pairwise comparisons against SDEdit, and perform 100 such evaluation tasks. The reward per task is kept as 0.2$. Since each task takes around min, the wage around 12$ per hour. For each question, the workers will be shown baseline model using the same input. The instruction is: "Which a better polished image for the input? An ideal polished image should look realistic, and matches the input in visual appearance (e.g., they look like the same person, with matched hairstyles and similar glasses)".
images: generated image from SDEdit and the other from the
26

---

Published as a conference paper at ICLR 2022
Painting
Diverse outputs
Figure 21: Stroke-based image generation bedroom images with SDEdit (VP) pretrained on
LSUN bedroom.
27

---

Published as a conference paper at ICLR 2022
Original
Input
Edited
Figure 22: Stroke-based image editing bedroom images with SDEdit (VP) pretrained LSUN bedroom. SDEdit generates image edits that are both realistic and faithful (to the user edit), while avoids making undesired modifications on pixels not specified by users
Add
Perturb with SDF
scribble
Original
Edited
Input
n=500
User edit image
Perturb image with SDE
n-500
n--00
D=300
n=200
100
n=
(c) Reverse SDE process
Figure 23: Stroke-based image editing. (a) Given image, users will first modify the image using stroke, and provide mask which describes the pixels covered by stroke. (b) The edited image will then fed into SDEdit. SDEdit will first perturb the image with an SDE, and then simulate the reverse SDE (see Algorithm 5). (c) We provide visualization of the intermediate steps of reversing SDE used in SDEdit.
28

---

Published as a conference paper at ICLR 2022
Original
Inpul
Edited
Figure 24: Stroke-based image editing on CelebA-HQ images with SDEdit. SDEdit generates image edits that are both realistic and faithful (to the user edit), while avoids making undesired modifica- tions on pixels not specified by users.
Original
Input
Edited
Figure 25: Image compositing on CelebA-HQ images with SDEdit. We edit the images to have brown hair. The model is pretrained on FFHQ.
29

---

Published as a conference paper at ICLR 2022
Original
Input
Edited
Figure 26: Image compositing on CelebA-HQ images with SDEdit. We edit the images wear glasses. The model is pretrained on FFHQ.
Original
Inpot
Edited
Figure 27: Image compositing on CelebA-HQ images with SDEdit. We edit the images to have blond hair. The model is pretrained on FFHQ.
30

---

Published as a conference paper at ICLR 2022
(a) Original dataset image.
(b) User edited input.
(c) SDEdi results.
Figure 28: Image compositing results with SDEdit (VE) on CelebA-HQ (resolution 1024x 1024). The SDE model is pretrained on FFHQ
Original dataset image.
(b) User edited input.
SDEdi results.
Figure 29; Stroke-based image editing results with SDEdit (VE) on CelebA-HQ (resolution
1024x 1024). The SDE model is pretrained on FFHQ.
Ground tuth
Simmlated stroke
StykGANN-AAA
In-domain
In-domain
e4c
SDEdit
Figure 30: Stroke-based image generation with simulated stroke paintings inputs on bedroom images
with SDEdit (VP) pretrained on LSSN hedroom dataset.
31

---

Published as a conference paper at ICLR 2022
Ground truth
Simulated stroke
StylGGAN2-ADA
e4c
SDEdit
Figure 31: Stroke-based image generation with simulated stroke paintings inputs on church images with SDEdit (VP) pretrained on LSUN church outdoor dataset.
Ground truth
Simulated stroke StyleGANZ-ADA
In In-domain
In-domain
e4e
SDEdit
Figure 32: Stroke-based image generation with simulated stroke paintings inputs on human face images with SDEdit (VP) pretrained on CelebA dataset.
Stroke painting
fg
to
fg
to 0.5
bn 0.6
4 0.7
to 0.8
0.9
to = 1.0
Figure 33: Trade-off between faithfulness and realism shown with stroke-based image generation with simulated stroke painting inputs on church images with SDEdit (VP) pretrained on LSUN church outdoor dataset.
32

---

Published as a conference paper at ICLR 2022
Painting
Tabby
Jaguar
Lion
Tiger
Painting
German shepherd
Boxer
Red fox
Coyote
Figure 34: Class-conditional image generation from stroke paintings with different class labels by
SDEdit (VP) pretrained on ImageNet.
Painting
Output
Cat
Horse
Figure 35: troke-bbaee image generation with stroke inputs on cat and horse images with SDEdit (VP) pretrained LSUN cat and horse dataset. Notice that for coarser guide (e.g. the third row), we choose to slightly sacrifice faithfulness order to obtain more realistic images by selecting a larger to 0.6, while all the other images generated with to 0.5.
33


Reviews:
[{"review_id": "oAQ74LszGX", "reviewer": "Reviewer_CnCY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe whole idea is interesting, effective and simple. The SDEdit perturbs the input image with the forward SDE and generates edits by reversing the SDE, the SDE is pretrained on unlabeled data, so the SDEdit does not require collecting pair images. Meanwhile, since the excellent generation performance of SDE, the SDEdit can generate the content with high quality. The mask strategy can help the user edit the regions where they want.\n\nWeaknesses:\nThe SDEdit can not edit the image flexibly enough, such as closing eyes, opening mouth, changing lip color. Meanwhile, the guidance can not influence the results well sometimes (Fig.1 Stroke-based Editing), there is no brown forest in the region where the user draws the brown stroke. Finally, the Stroke-based Editing is not complex and some methods can do this such as [1][2], the SDEdit should compare with them.\n\n[1]DeFLOCNet: Deep Image Editing via Flexible Low-level Controls. CVPR2021\n\n[2]Sc-fegan: Face editing generative adversarial network with user’s sketch and color. ICCV 2019", "summary_of_the_paper": "The paper proposesed an image editing and synthesis system (SDEdit) with SDE. By injecting the guided image into the reverse process with appropriate $t_0$, the results can have a trade-off between realism and faithfulness.", "summary_of_the_review": "The paper is well-written and easy to follow. The results are impressive. Although Stroke-based Editing is not complex, especially on the face,  I think this is a good start of using the SDE to do the image editing task. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QSlk84uhMj", "reviewer": "Reviewer_Hvki", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The SDE formulation of image editing is an interesting approach, it's not only able to generate image with realism (through diffusion process) but also capture the real input (by controlling the time variable t0 that runs the diffusion process).\n\n2. The proposed method is able to produce a diverse set of predictions given user input by sampling different Gaussian noise in the process. \n\n3. The paper is well written and easy to follow.\n\nWeakness\n\n1. The paper is overclaimming. It is argued that the proposed method is able to diffuse the image \"given a user guide of any type\", however, it seems the proposed method is only able to handle the user guide in a form of manipulating pixel rgb values (strokes, color patch, or image patch), there are many other user guides exist, such as semantic mask (GauGAN), or text (using CLIP model), and it's not clear how the proposed method generalize to these user guides.\n\n2. The proposed method need to run iteratively to obtain the final image, which makes the inference time inevitably long. It would be great to include the running time of the proposed method in comparing with other methods. \n\n3. The main technique in the paper (The SDE process) is previous method (Song et. al. 2020, 2021), the paper only proposed to use the human edited image as initialization of SDE at a particular time t0, thus the novelty is not great enough. ", "summary_of_the_paper": "The paper proposed Stochastic Differentiable Image Editing method which iteratively diffuse the human edited image  to real images, using the pretrained score-based SDE models (the edit is in a form of manipulating image pixel values, e.g. stroke, color paint, or image patch).The method achieves both realistic image editing and faithfulness to human input through user studies. ", "summary_of_the_review": "The paper proposed an interesting technique to edit image using the SDE. The generated image is not only faithful but also relistic. However, considering the technique novelty and the overclaiming, I vote for a weak reject initially and am willing to listen to the authors and other reviewers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GJLCf3_aX2z", "reviewer": "Reviewer_N9v6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Stroke-based image editing algorithms have been widely-adopted and popular for many artistic design tools and softwares. Many of those are (conditional) GAN-based methods, and can be often challenging to balance the tradeoff between use control vs. result faithfulness. \n\nThe proposed method has a novel technical contribution, that utlizes iterative SDE denoising for guided stroke sketch input + perturbed noise, with photorealistic output via the reverse of the SDE. The paper has compared with a few state-of-the-art baselines and demosntrate its effectiveness in producing more faithful editing results.\n\nOne question I have is that, how good the guided stroke sketch needs to be? I don't see ablation study on this, but I think it is an important factor for practical applications. \n\nPaper writing and references seem fine to me. ", "summary_of_the_paper": "This paper proposes to synthetize and edit realistic images based on stocahstic differential equations (SDE). The method is based on diffusion generative model that iteratively denoises through SDEs and demosntrates that this SDE prior increases photorealism of the images, compared to state-of-the-art GAN-based approaches. ", "summary_of_the_review": "Overall I am leaning towards a weak accept. The paper has clear novelty regarding stroke-based image editing via perturb + reverse SDEs. Comparisons and ablation study look good to me except for the one on the level of user sketch control. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to ICLR.

This paper presents a technique for image synthesis based on stochastic differential equations and a diffusion model.  This looks to be a very nice idea with good results.  After discussion, the reviewers converged and all agreed that the paper is ready for publication---the most negative reviewer raised their score after the author rebuttal, from a weak reject to weak accept.  The rebuttal clearly and concisely addressed several concerns of the reviewers.

I'm happy to recommend accepting the paper.
Decision: Accept (Poster)