DiffEdit: Diffusion-based semantic image editing with mask guidance
Authors: Guillaume Couairon
Jakob Verbeek
Holger Schwenk
Matthieu Cord
Venue: ICLR 2023 notable top 25% (2023)
Keywords: computer vision
image editing
diffusion models

Abstract: Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. 
Current editing methods based on diffusion models usually require to provide a mask, making the task much easier by treating it as a conditional inpainting task. In contrast, our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask-based diffusion. 
DiffEdit achieves state-of-the-art editing performance on ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset as well as text-based generated images.

Published as a conference paper at ICLR 2023
DIFFEDIT: DIFFUSION-BASED SEMANTIC IMAGE EDIT- ING WITH MASK GUIDANCE
Guillaume Couairon, Jakob Verbeck, Holger Schwenk
Matthieu Cord
Meta AI
Sorbonne Université, Valco.ai matthieu. corde sorbonne-universsine
{gcouairon, iverbeek, schwenk @meta attmeta.com com com
ABSTRACT
Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. this article, propose DIFFEDIT, method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal to edit image based text query. Semantic image editing is extension of image generation, with the additional constraint that the generated image should as similar possible given input image. Current editing methods based on diffusion models usually require to provide mask, making the task much easier conditional inpainting task. In contrast, our main contribution able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask- based diffusion. DIFFEDIT achieves sfute-of-the-at editing performance ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset well text-based generated images.
by treating
Input Image
Masked Diffusion
laput Image
Masked Diffusion
Encode
Encade
Caplin:: hovel of fruits Query: A of pours
Generate Mask
Caption: Banri of fraits Query: basket of fiuits
Gemerate Mask
Figure 1: In semantic image editing the goal is to modify input image based on textual query, while otherwise leaving the image as close as possible to the original. In our DIFFEDIT approach, a mask generation module determines which part of the image should be edited, and an encoder infers the latents, to provide inputs to a text-conditional diffusion model which produces the image edit.
INTRODUCTION
The task of semantic image editing consists in modifying an input image in accordance with textual transformation query. For instance, given an image bowl of fruits and the query "fruits" - "pears", the aim is to produce novel image where the fruits have been changed into pears, while keeping the bowl and the background as similar as possible to the input image. The text query can also be more elaborate description like "A basket of fruits". See the example edits obtained with DIFTEDIT in Figure I. Semantic image editing bears strong similarities with image generation and can be viewed extending cext-conditional image generation with an additional constraint: the generated image should be as close as possible to given input image.
Text-conditional image generation is currently undergoing revolution, with DALL-E (Ramesh et al., 2021), Cogview (Ding et al., 2021), Make-a-scene (Gafni et al., 2022), Latent Diffusion Mod- els (Rombach et al., 2022), DALL-E (Ramesh et al., 2022) and Imagen (Saharia al., 2022b),

---

Published as a conference paper at ICLR 2023
vastly improving state of the art in modelling wide distributions of images and allowing for un- precedented compositionality of concepts in image generation. Scaling these models is a key to their success. State-of-the the art models are trained vast amounts of data, which requires large computational resources. Similarly to language models pretrained on web-scale data and adapted in downstreams tasks with prompt engineering. the generative power of these big generative mod- els can be harnessed to solve semantic image editing, avoiding to train specialized architectures (Li al., 2020a; Wang et al., 2022a), or use costly instance-based optimization (Crowson al., 2022; Couairon et al., 2022; Patashnik et al., 2021).
Diffusion models are an especially interesting class of model for image editing because of their it- erative denoising process starting from random Gaussian noise. This process can be guided through a variety of techniques, like CLIP guidance (Nichol et al., 2021; Avrahami et al., 2022; Crowson, 1021), and inpainting by copy-pastinng pooy-pastnng pixel values outside a user-given mask (Lugmayr al., 2022).
These previous works, however, lack two crucial properties for semantic image editing: (i) inpaint- ing discards information about the input image that should be used in image editing (e.g. changing a dog into a cat should not modify the animal's color and pose); (ii) a mask must be provided as input to tell the diffusion model what parts of the image should be edited. We believe that while drawing masks is common on image editing tools like Photoshop, language- -ugdedd editing offers a more intuitive interface modify images that requires less effort from users.
Conditioning a diffusion model on an input image can also be done without mask, e.g. by consid- ering the distance input image as a loss function (Crowson, 2021; Choi et al., 2021), )), or by using a noised version of the input image as starting point for the denoising process as in SDEdit (Meng al., 2021). However, these editing methods tend to modify the entire image, whereas we aim for localized edits. Furthermore, adding noise to the input image discards important information, both inside the region that should be edited and outside.
To leverage the best of both worlds. we propose DIFFEDIT, an algorithm that leverages a pretrained text-conditional diffusion model for zero-shot semantic image editing, without expensive editing- specific training. DIFFEDIT makes it possible by automatically finding what regions of an input image should be edited given a text query, by contrasting the predictions of conditional and uncon- ditional diffusion model. We also show how using reference text describing the input image and similar to the query, can help obtain better masks. Moreover, we demonstrate that using a reverse denoising model, to encode the input image in latent space, rather than simply adding noise it, allows to better integrate the edited region into the background and produces more subtle and natural edits. See Figure for illustrations. We quantitatively evaluate our approach and compare to prior work using images of the ImageNet and COCO dataset, as well as a set of generated images.
RELATED WORK
Semantic image editing. The field of image editing encompasses many different tasks, from photo colorization and retouching (Shi et al., 2020), to style transfer (Jing ct al., 2019). inserting objects in images (Gafni & Wolf, 2020; Brown et al., 2022), image-to-imade translation (Zhu et al., 2017; Sa- haria et al., 2022a), inpainting (Yu et al., 2018), scene graph manipulation (Dhamo et al., 2020), and placing subjects in novel contexts (Ruiz et al., 2022). We focus on semantic image editing, where the instruction to modify an image given in natural language. Some approaches involve training an end-to-end architecture with proxy objective before being adapted to editing at inference time, based on GANs (Li et al., 2020b;a; Ma et al., 2018; Alami Mejjati et al., 2018; Mo al., 2018) or transformers ( (Wang et al., 2022a; Brown et al., 2022; Issenhuth et al., 2021). Others (Crowson et al., 2022: Couairon et al., 2022: Patashnik et al., 2021; Bar-Tal et al., 2022) rely on optimization of the image itself, or a latent representation of it, to modify an image based on a high-level multimodal objective in an embedding space, typically using CLIP (Radford et al., 2021). These approaches quite computationnaly intensive, and work best when the optimization is coupled with a pow- erful generative network. Given a pre- pre-trained generative model such as a GAN, it has also been explored to find directions in the latent space that corresponds to specific semantic edits (Härkönen et al., 2020; Collins et al., 2020; Shen et al., 2020; Shoshan et al., 2021), which then requires GAN inversion to edit real images (Wang et al., 2022c; Zhu et al., 2020; Grechka et al., 2021).
Image editing with diffusion models. Because diffusion models iteratively refine an image starting from random noise, they are easily adapted for inpainting when mask is given as input. Song al.
2

---

Published as a conference paper at ICLR 2023
(2021) proposed condition the generation process by copy-pasting pixel values from the reference image at each denoising step. Nichol et al. (2021) use a similar technique by copy- oopy-pasting pixels in the estimated final version of the image. Wang al. (2022b) use DDIM encoding of the input image, and then decode on edited sketches or semantic segmentation maps. The gradient of a CLIP score can also be used to match given text query inside mask, in Paint by Word (Bau et al., 2021), local CLIP-guided diffusion (Crowson, 2021), or blended diffusion (Avrahami et al., 2022). Lug- mayr et al. (2022) apply sequence of noise-denoise operations to better inpaint a specific region. There are also number of methods that do not require an editing mask. In DiffusionCLIP (Kim & Ye, 2021), the weights of the diffusion model themselves are updated via gradient descent from CLIP loss with target text. The high computational cost of fine-tuning a diffusion model for each input image, however, makes impractical as an interactive image editing tool. In SDEdit (Meng et al. 2021) the image is corrupted with Gaussian noise, and then the diffusion network is used to denoise it. While this method originally designed to transform sketches to real images and to make pixel-based collages more realistic, text query. In ILVR (Choi et al., 2021), the decoding process of diffusion model is guided with the constraint that downsampled versions of the input image and decoded image should stay close. Finally, in recent work concurrent to ours, Hertz et al. (2022) propose to edit images by modifying attention maps during the diffusion process.
adapt it by denoising the image conditionally to the
DIFFEDIT FRAMEWORK
In this section, we first give overview of diffusion models. We then describe our DIFFEDIT approach in detail, and provide a theoretical analysis comparing DIFFEDIT with SDEdit.
3.1 BACKGROUND: DIFFUSION MODELS, DDIM AND ENCODING
Denoising diffusion probabilistic models (Ho et al., 2020) is class of generative models that are trained to invert a diffusion process. For a number of timesteps T, the diffusion process gradually adds noise to the input data, until the resulting distribution is (almost) Gaussian. A neural network is then trained to reverse that process, by minimizing the denoising objective
(1) where eg is the noise estimator which aims to find the noise E ~ NO, I) that is mixed with an input image Xq to yield x, Vo,X0 /1 are. The coefficient oa defines the level of noise and is decreasing function of the timestep 1, with og (no noise) and or z 0 (almost pure noise).
L Exx,,,,l|e (x, t)l2:
Song et al. (2021) propose to co to generate new images with the DDIM algorithm: starting from XT ~ (O, I), the following update rule is applied iteratively until step O:
vI
Cgep (Xe, 1)
x -
/1 or-
Voi
The variable is updated by taking small steps in the direction of eg. Equation 2 can be written as the neural ODE, taking u = x/ Va and T / 11/a 1:
u
du = E VI+ t)dT.
(3)
This allows to view DDIM sampling as an Buler scheme for solving Equation 3 with initial condition u(I=T) ~ _N(0, I). This illustrates that can use fewer sampling steps during inference than the value of 7 chosen during training, by using a coarser discretization of the ODE. In the remainder of the paper, we parameterize the timestep t be between 0 and so that
corresponds to T steps of diffusion in the original formulation. As proposed by Song et al. (2021), we can also use this ODE to encode an image Xa onto a latent variable x, for timestep r S 1, by using the boundary condition u(t 0) Xo instead of u(t = 1). and applying an Euler scheme until timestep r. In the remainder of the paper, we refer to this encoding process as DDIM encoding, we denote the the encoding ratio.
corresponding function that maps Xo to x, as Er, and refer to the variable Similarly, we note D, the inverse function that maps x, to Xg, which corresponds to regular DDIM decoding. With sufficiently small steps in the Euler scheme, decoding x, approximately recovers the original image Xn- This property is particularly interesting in the context of image editing: all the information of the input image Xo is encoded in X,, and can be accessed via DDIM sampling.
3

---

Published as a conference paper at ICLR 2023
Step Compute Mask
Estimate conditioned to reference text
nomellied difference
Gaussian
noise
Estimate noise conditionaly query
Query Q: Zehra
Step 2: Encode with DDIM until encoding ratio
Ref.
Horse
Text
Reveric DRIM
Revense DDIM
X1
X+
Step 3: Decode with mask-wise correction
May (1
Output
DDIM
DDIM
DOIM
Skep
Step
Step
sefra
zeóra1
Figure 2: The three steps of DIFFEDIT. Step I: add noise to the input image, and denoise it: once conditioned on the query text, and once conditioned on reference text (or unconditionally) We derive mask based on the difference in the denoising results. Step 2: we encode the input image with DDIM, to estimate the latents corresponding to the input image. Step 3: we perform DDIM decoding conditioned on the text query, using the inferred mask to replace the background with pixel values coming from the encoding process the corresponding timestep.
3.2 SEMANTIC IMAGE EDITING WITH DIFFEDIT
In many cases, semantic image edits can be restricted to only part of the image, leaving other parts unchanged. However, the input text query does not explicitly identify this region, and a naive method could allow for edits all over the image, risking to modify the input areas where it To circumvent this, we propose DIFFEDIT, a method to leverage a text-conditioned diffusion model infer mask the region that needs to be edited. Starting from DDIM encoding of the input image, DIFFEDIT uses the inferred mask to guide the denoising process, minimizing edits outside the region of interest. Figure 2 illustrates the three steps of our approach, which we detail below.
needed.
Computing editing mask. When the denoising an image, a text-conditioned diffusion model will yield different noise estimates given different text conditionings. We can consider where the estimates are different, which gives information about what image regions are concerned by the change in conditioning text. For instance, in Figure 2, the noise estimates conditioned to query zebra and reference text horse' are different on the body of the animal, where they will tend to decode different colors and textures depending on the conditioning. For the background, on the other hand, there is little change in the noise estimates. The difference between the noise estimates can thus be used to infer a mask that identifies what parts on the image need to be changed to match the query. In algorithm, Appendix 1) remove extreme values in noise predictions and stabilize the effect by averaging spatial differences over set of input noises, with 10 in our default configuration. The result then rescaled to the range [0, 1]. and binarized with a threshold, which we set to 0.5 by default. The masks generally somewhat overshoot the region that requires editing, this beneficial as it allows to be smoothly embedded in it's context, see examples in Section 4 and Section A.5.
Step 1:
Gaussian noise with strength 50% (see analysis in
Step 2: Encoding, We encode the input image Xo in the implicit latent space at timestep r with the DDIM encoding function E,. This is done with the unconditional model, i.e. using conditioning text 0, so no text input is used for this step.
We also empty reference text, which denote as Q
4

---

Published as a conference paper at ICLR 2023
Step 3: Decoding with mask guidance. After obtaining the latent x,,, decode it with our diffu- sion model conditioned on the editing text query Q. e.g. zebra e.g.zebra in the example of Figure? 2. We use our mask M to guide this diffusion process. Outside the mask M, the edited image should in principle be the same as the input image. We guide the diffusion model by replacing pixel values outside the mask with the latents x; inferred with DDIM encoding, which will naturally map back to the original pixels through decoding, unlike when using a noised version of X as typically done (Meng et al., 2021; Song et al., 2021). The mask-guided DDIM update can be written as y, My (1 M )x., where is computed from dt with Eq. 2, and x; is the corresponding DDIM encoded latent.
The encoding ratio r determines the strength of the edit: larger values of allow for stronger edits that allow to better match the text query, at the cost of more deviation from the input image which might not be needed. We evaluate the impact of this parameter in our experiments. We illustrate the effect of the encoding ratio in Appendix A.5.
3.3 THEORETICAL ANALYSIS
In DIFFEDIT, we use DDIM encoding to encode images before doing the actual editing step. this section, we give theoretical insight on why this component yields better editing results than adding random noise as in SDEdit (Meng ct al., 2021). With x, being the encoded version of Xo. using x, unconditionally would give back the original image Xo- In DIPTEDIT, we use DDIM decoding conditioned on the text query Q, but there is still a strong bias to stay close to the original image. This is because the unconditional and conditional noise estimator networks Eg and (. Q) often produce similar estimates, yielding similar decoding behavior when initialized with the same starting point x,. This means that the edited image will have a small distance w.r.t. the input image, a property critical in the context of image editing. We capture this phenomenon with the proposition below, where we compare the DDIM encoder (xo) the SDEdit encoder (xx0, e) := Va,X0 + vI - a,e, which simply adds noise to the image Xy.
DDIM decoding
Proposition 1. Let X = Rdd be the space of input images, PD be the data distribution of couples (Xo, Q) where Xo E X and Q a textual query to edit that image. Suppose that [|ea(x,, 2,1) S C' for all x E X, t E [0, 1], that ea(-, 0, t) is K Ki-Lipschitz for all t, and let K₂ = Ex E)pp max [|co(x, Q.t) co(x, 0, t)ll- Then, for all encoding ratios S r $ 1,
we have the two following bounds:
E (xopQ)<p c-N(0.1)
I (, e). 1)112 s (C +C+1)).
(4)
[xo (x), Q1112 S VF +I T V 72 +1) K₁
K2T
E
(5)
(xa.Q) Qrepn
where 7 = 1//0, increases with the encoding ratio r: 7(r 0) 0 and lim == +00.
We provide the proof in Appendix B. The first bound is associated with SDEdit, and is extension of a bound proven in the original paper. The second bound we con- tribute is associated with DIFFEDIT. It is tighter than the first bound below a certain encoding ratio, see Figure 3. We empirically estimated the parameters K1, K2 and C with the diffusion models that we are using. While the
SDEdit upper bound
L2 Distance
DiftEdit upper bound
asymptotic behavior of the second bound worse than the first with K₃ > 1, it is the very small value of K₂ that gives a tighter bound.
0-
0%
20%
40%
60% 80%
This supports our argument from above: because the un- conditional and text-conditional noise estimates generally give close results -K₃ being a measure of the average difference- the Euler scheme with ca(, 2. gives a se- quence of intermediate latents yr..... .... Yo that stays close to the trajectory æy...., D,,,,, ~ x. mapping back x, to Xp. While these upper bounds do not guarantee that DDIM encoding yields smaller edits than SDEdit, exper-
Encoding ratio r
Figure 3: Illustration of the bounds from Proposition 1, with estimated pa- rameters C= 1, =000., and K₁ =3.
imentally we find that it is indeed the case.
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section, we describe our experimental setup, followed by qualitative and quantitative results.
4.1 EXPERIMENTAL SETUP
Datasets. We perform experiments on three datasets. First, on ImageNet (Deng et al., 2009) we follow the evaluation protocol of FlexIT (Couairon et al., 2022). Given an image belonging to one class, the goal is to edit so that it will depict an object of another class as indicated by the query. Given the nature of the ImageNet dataset, edits often concern the main object in the scene. Second, we consider editing images generated by Imagen (Saharia et al., 2022b) based on structured text prompts, in order to evaluate edits that involve changing the background, replacing secondary objects, or changing object properties. Third, we consider edits based on images and queries from the COCO (Lin et al., 2014) dataset to evaluate edits based more complex text prompts.
Diffusion models. In our experiments we use latent diffusion models (Rombach et al., 2022). We the clnss-conditional model trained on ImageNet resolution 256 256, as well the 890M parameter text-conditional model trained on LAION-5B (Schuhmann et al., 2021), known as Stable Diffusion, 512 x 512 resolution. Since these models operate in VQGAN latent spaces (Esser et al., 2021), the resolution of our masks is 32 x 32 (ImageNet) or 64 x 64 (Imagen and COCO). We use 50 steps in DDIM sampling with fixed schedule, and the encoding ratio parameter further decreases the number of updates used for our edits. This allows to edit images in 10 seconds on a single Quadro GP100 GPU. We also use classifier-free guidance (Ho & Salimans, 2022) with the recommended values: 5 on ImageNet, 7.5 for Stable Diffusion. For more details see Section A.2.
Comparison other methods. We use SDEdit (Meng et al., 2021) as our main point of compari- son, since we can use the same diffusion model as for DIFFEDIT. We also compare to FlexIT (Coua- iron et al., 2022), a mask-free, optimization- based editing method based on VQGAN and CLIP. On ImageNet, we evaluate ILVR (Choi et al., 2021) which uses another diffusion model trained on ImageNet (Dhariwal Nichol, 2021). Finally, on COCO and Imagen images, we compare to the concurrent work of Hertz et al. (2022).
Evaluation. In semantic image editing, we have to satisfy the two contradictory objectives of (i) matching the text query and (ii) staying close to the input image. For given editing method, better matching the text query comes at the cost of increased distance to the input image. Different editing methods often have parameter that allows to control the editing strength: varying its value allows to get different operating points, forming a trade-off curve between the two objectives aforementioned. Therefore, we evaluate editing methods by comparing their trade-off curves. For diffusion-based methods, we use the encoding ratio to control the trade-off.
4.2 EXPERIMENTS ON IMAGENET
Copy
On ImageNet, we follow the evaluation protocol of Coua- iron et al. (2022), with the associated metrics: the LPIPS perceptual distance (Zhang et al., 2018) measures the dis- tance with the input image, and the CSFID, which is class-conditional FID metric (Heusel et al., 2017) mea- suring both image realism and consistency w.r.t. the trans- formation prompt. For both metrics, lower values indicate better edits. For more details see Couairon et al. (2022).
100
Diffdit
SDEdit
CSFID
75
ILVR
FlexIT
50%
60%
90%
Retrieve
20
40
60
RD
LPIPS
We compare DIFFEDIT to other semantic editing meth- ods from the literature in terms of CSFID-LPIPS trade- off. Stronger edits improve (lower) the CSFID score as the edited images better adhere to the text query, but the resulting images tend to deviate more from the input im- age, leading to worse (increased) LPIPS distances.
Figure 4: Comparison on ImageNet data of DIFFEDIT with other Image Editing methods. For DIFFEDIT we annotate the different operating points with the corresponding encoding ratios.
2 Availlabl at https: / //huggi a ce. co/Cooooooo co//ompvi 8 h/staallleddeeeeeeeeeee ffus on.
3 As there is official implementation available at the time of writing, we used the unofficial implementa- tion adapted for Stable Diffusion from https: //gi thu..hhu....... con/bl oc wwtp///o////cc/////o//// Crtsssttesttttttccecceoo onControl.

---

Published as a conference paper at ICLR 2023
Iaput Image
Text Query
English fochound
Ambulance
Axoled
Bald cugle
Black grouse
Bixer
Brown bear
Zabra
SDEdit
Encode- Decode
Our Mask
(before binarization)
DiflEii
wio Encode
DiffEdit
Figure 5: Edits obtained on ImageNet with DIFTEDIT and ablated models. DIFFEDIT without masking, and SDEdit is obtained when not using masking nor encoding. When not using masking (SDEdit and Encode-Decode) we observe undesired edits to the background, see e.g. the sky in the second column. When not using DDIM encoding (SDEdit and DiffEdit w/o Encode), appearance information from the input -such posc- is lost, see last two columns.
is
The results in Figure 4 indicate that DIFFEDIT obtains the best trade-offs among the different meth- ods. For fair comparison with previous methods, here we do not leverage the label of the input image and use the empty text reference when inferring the editing mask. The Copy and Retrieve baselines are two opposite cases where we have best possible LPIPS distance -zero, by copying the input image- and best possible transformation score by discarding the input image and replac- ing it with a real image from the target class from the ImageNet dataset. DIFFEDIT, well as the diffusion-basee SDEdit and ILVR, are able obtain CSFID values comparable that of the retrieval baseline. Among the diffusion-based methods, our DIFFEDIT obtains comparable CSFID values significantly better I.PISS scores. For FlexITT, the CSFID best value is significantly worse, indicating it is not able to produce both strong and realistic edits. Using more optimization steps does not solve this issue, as the distance to the input image is part of the loss minimizes.
Ablation experiments. We ablate the two core components of DIFFEDIT, mask inference and DDIM encoding, to measure their relative contributions in terms of CSFID-LPIPS trade-off. If we do not either of these components Figure 6, left panel, show that adding DDIM encoding Sincode-Decodee and the masking (DiffEdit w/o Encode) separately both improve the trade-off and reduce the average editing distance input image compared to SDEdit. Moreover, combining these two elements into DIFFEDIT gives an even better trade-off, showing their complementarity: masking better preserves the background, while DDIM encoding retains visual appearance of the input inside the mask. See Figure for qualitative examples of these ablations, along with the inferred masks.
method reverts to SDEdit (Meng et al., 2021). The results in
the
The right panel of Figure 6 shows DIFFEDIT with different mask bi- narization thresholds. Compared to our default value of 0.5. lower threshold of 0.25 results in larger masks (more image modifications) and worse CSFID-LPIPS tradeoff. A higher threshold 0.75 results in masks that too restrictive: the CS- FID score stagnates around 40, even large encoding ratios.
Thros 0.5
SDEdit
* Thros. 0.25
CSFID
60-
weln Fnende
Diflk dit
40
Encode-Decode
20
40
60
20
40
LPISS
LPIPS
Figure 6: Ablations ImageNet. Left: effect of masking and encoding component. Right: DIFFEDIT with different mask thresholds; with 0.5 our default setting.
7

---

Published as a conference paper at ICLR 2023
Finally, mask guidance operator y. My: (1 M)x, provides better trade-off than the operator used in GLIDE (Nichol et al., 2021), which interpolates y: with a mask-corrected version of the predicted denoised image Yo. With encoding ratio 80%, both operators produce edits with LPIPS score of 30.5, but the GLIDE version yields CSFID of 26.4 compared to 23.6 for ours.
4.3 EXPERIMENTS ON IMAGES GENERATED BY IMAGEN
In our second set of experiments we evaluate edits that involve changes in background. replacing secondary objects, and editing object properties. We find that images generated by Imagen (Saharia et 2022b) offer well suited testbed for this purpose. Indeed, the authors tested the compositional abilities of Imagen with templated prompts of the form: "(A photo of An oil painting of a} {fuzzy panda British shorthair cat Persian cat Shiba Inu dog raccoon) {wearing a cowboy hat and wearing sunglasses and} {red shirt black jacket) {playing a guitar riding a bike skateboarding) {in a garden on beach | on top of a mountain) " resulting in 300 prompts. We use the generated images in- put and ask to change the prompt
80%
90%
Diffi
another prompt for which one of these elements is changed. Since we cannot the CSFID metric as for ImageNet, as images do not carry a single class label, use FID to measure image realism, and CLIP-
30
70%
70%
Diffaaie
woree
CLIPScore
GO
wr ref lext
SDElit
50%
29
Encode
Dacod:
40
Choss
Copy
Control
30
Score (Hessel et al., 2021) to mea- sure the alignment of the query and output image. These two scores have become the standard in evalu- ating text-conditional image genera- tion (Saharia et al., 2022b).
LPIPS
CLIPScoree
Figure Editing trade-offs on Imagen images.
Input Image
Query
wio Ref Text
Ref. Text
Afoell frale
banket
Figure displays the CLIP-LPIPS and FID-CLIP trade-offs. DIFFEDIT provides more accurate edits than SDEdit, FlexIT, and Cross Atten- tion Control, by combining inferred masks with DDIM encoding. Two versions of DifiEdit are shown, which differ by how the mask is com- puted: they correspond to (i) using the original caption as reference text (labelled w/ ref. text) or (ii) using the empty text 0 (labelled w/o ref. text).
offoren
Pa
besci
Mass
Outpst
Mask
Ouipar
Figure 8: Masks and edits obtained with and without refer- ence text in the mask computation algorithm.
Input Image
Computing the mask with the orig- inal caption as reference text yields the best overall trade-off. Leverag- ing the original caption yields better CLIP and FID scores. Figure il- lustrates the difference in the masks obtained with and without reference text for two examples. The refer- ence text allows ignore parts of the image that are described both by the query and reference text (e.g. the fruits), because in both cases the net- work uses the common text on the corresponding image region to esti- mate the noise. On the contrary, parts where the query and reference text disagree, e.g. "bowl" vs. "basker", will have different noise estimates.
Text Query (simplified)
combay sunglowce
sunglasses cuwboy
Shite dog
the mountain the
- playing the guite
SDEdit
(70%)
DiffEdit
DifTdiit (90%)
Figure 9: Edits on Imagen dataset. We use encoding ratio of 90% for DifffEdit and 70% for SDEdit for fair comparison: both methods have similar CLIPScore, for larger encoding ratios SDEdit drastically change the input.
8

---

Published as a conference paper at ICLR 2023
Qualitative transformation examples shown in Figure 9, where the masks inferred by trasting the caption and query texts.
4.4 EXPERIMENTS ON COCO
To evaluate semantic image editing with more complex prompts, we use images and captions from the COCO dataset Lin et al. (2014). To this end, we leverage the annotations provided by Hu et al. (2019), which associate images from the COCO validation set with other COCO captions that are similar to the original ones, but in contradiction with the given image. This makes these annotations particularly interesting as queries for semantic image editing, as they can often be satisfied by editing only part of the input image, see Figure 15 in the supplementary material for examples. Similar to our evaluation for Imagen images, here we evaluate edits in terms of CLIPScore, FID and LPIPS.
The results in Figure 10 show that the CLIP-LPIPS trade-off of DIFFEDIT is the best, but that reaches lower maximum CLIP score than SDEdit. The FID scores are similar to SDEdit, but significantly improves upon the Encode-Decone ablation, which does not Moreover, in contrast to results on the Imagen data, leveraging the original image caption does not change the CLIP-LPIPS and FID-CLIP trade- offs. We find that the caption often describes the input image differently compared to the query text, making it more difficult to identify which part of the image needs to be edited. We verify this hypothesis in Section A.3 by filtering the dataset according to the edit distance between the caption and edit query. When the caption and edit query are similar, leveraging the image caption boosts CLIP scores by 0.25 points, a similar improvement as seen on the Imagen data.
mask.
40.0
DidEat
90%
we ref, sext
30
37.5
CLIPScare
70%
29
35.0
SDEdi
Encode
50%
32.5
28
FlexTT
20%
30.0
Crossktt.
Control
27
20
40
60
27.5
30.0
I.IPS
CLIPScore
Figure 10: Quantitative evaluation COCO.
Imput Image
Qualitative examples are shown in Figure 11. The first column illus- trates the benefit of DDIM encod- ing: we are able correctly main- tain properties of the object inside the mask, such as clothes' color. The three last columns illustrate how con- trasting different pairs reference and query text allows to select differ- ent objects the input image to per- form different edits. See Section A.5 for more examples.
Text
A [skling wnwsbanning).
[Trea City bockground.
Living with [hros capot).
Living nom with [TTV paintig moutting).
ATV
[beown Gecplass)
DISTEd
Mask
DiTFdi
uutqui
Figure 11: Examples edits on COCO images.
5 CONCLUSION
We introduced DIFFEDIT, a novel algorithm for semantic image editing based on diffusion mod- els. Given a textual query, using the diffusion model, DIFFEDIT infers the relevant regions to be edited rather than requiring a user generated mask. Furthermore, in contrast to other diffusion-based methods, we initialize the generation process with DDIM encoding of the input. We provide the- oretical analysis that motivates this choice, and show experimentally that this approach conserves more appearance information from the input image, leading to lighter edits. Quantitative and qual- itative evaluations on ImageNet, COCO, and images generated by Imagen, show that our approach leads excellent edits, improving over previous approaches. Although DIFFEDIT works better with reference text describing the input image, we believe this additional information can be inferred from input image and target caption, which we leave for future work.
We

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Image editing raises several ethical challenges that we wish to discuss here. First, as image editing closely related to image generation, inherits known concerns. Open-source diffusion models are trained on large amounts of web-scraped data like LAION, and inherit their biases. In par- ticular, it was shown that LAION contains inappropriate content (violence, hate, pornography), along with racist and sexist stereotypes. Furthermore it was found that diffusion models trained on LAION, such as Imagen, can exhibit social and cultural bias. Therefore, the use of such models can raise ethical issues, whether the text prompt is intentionnally harmful or not. Because image edit- ing usually performed on real images, there are additionnal ethical challenges, such as potential skin tone change when editing person or re-inforcing harmful social stereotypes. We believe that open-sourcing editing algorithms in a research context contributes to a better understanding such problems, and aid the community in efforts to mitigate them in the future. Furthermore, image editing tools could used with harmful intent such as harrassement or propagating fake news. This use, known deep fakes, has been largely discussed in previous work, e.g. Btienne (2021). To mitigate potential misuse, the Stable Diffusion model is released under license focused on ethical and legal use, stating explicitly that users "must not distribute harmful, offensive, dehumanizing content or otherwise harmful representations of people or their environments, cultures, religions, etc. produced with the model weights".
Our editing benchmark based on the COCO dataset also has some limitations. COCO has pre- dominant western cultural bias, and we are therefore evaluating transformations on small subset of images mostly associated with western culture. Finding relevant transformation prompts for an image is challenging: while we found relevant to leverage existing annotations based on COCO, we believe that evaluating image editing models on a less culturally biased dataset is needed.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. Unsupervised attention-guict guided image-to-image translation. Advances neural information pro- cessing systems, 31, 2018.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Tex12LIVE: Text- driven layered image and video editing. arXiv preprint, arXiv:2204.02491, 2022.
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint, arXiv:2103. 10951, 2021.
Andrew Brown, Cheng-Yare Yang Fu, Omkar Parkhi, Tamara L Berg, and Andrea Vedaldi. End-to-end visual editing with generatively pre-trained artist. arXiv preprint, arXiv:2205.01666. 2022.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: 2021.
Conditioning method for denoising diffusion probabilistic models. In 2C2V,
Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of GANs. In CVPR, 2020.
Guillaume Couairon, Asya Grechka, Jakob Verbeck, Holger Schwenk, and Matthieu Cord. FlexIT: Towards flexible semantic image translation. In CVPR, 2022.
Katherine Crowson. CLIP Guided Diffusion HQ 512x512. 2021. URL https://colab. wtt.:///TTTTTTTTTTTTITTATAA
research. google..oooooooooooooooooooo.... com/
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castri- cato, and Edward Raff. VQGAN-CLIP. Open domain image generation and editing with natural language guidance. In ECCV, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D Hager, Federico Tombari, and Christian Rupprecht. Semantic image manipulation using scene graphs. CVPR, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Vog Mastering text-to-image generation via transformers. In NeurlPS, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Hubert Etienne. The future of online trust (and why deepfake is advancing it). Al and Ethics, 1, 11 2021. doi: 10.1007/:43681-0211
Oran Gafni and Lior Wolf. Wish you were here: Context-aware human generation. In CVPR, 2020.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
Asya Grechka, Jean-François Goudou, and Matthieu Cord. MAGECally invert images for realistic editing. In BMVC, 2021.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering interpretable GAN controls. In NeurIPS, 2020.

---

Published as a conference paper at ICLR 2023
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to-prome image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. arXiv preprint, arXiv:2104.08716 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge a local Nash equilibrium. In NeurIPS, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12595 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Hexiang Hu, Ishan Misra, and Laurens van der Maaten. Evaluating ext-to-image matching using binary image selection (BISON). In ICCV Workshop on closing the loop between vision and language, 2019.
Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard. EdiBERT, generative model for image editing. arXiv preprint, arXiv:2111.15264 2021.
Yongeheng Jing, Yezhou Yang. Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. Transactions visualization and computer graphics, 26(1 36:31):3365- 3385, 2019.
Valentin Khrulkov and Ivan Oseledets. Understanding DDPM latent codes through optimal trans- port. Applied Mathematics Letters, 2022.
Gwanghyun Kim and Jong Chul Ye. DiffusionCLIP: Text-guided image manipulation using diffu- sion models. CVPR, 2021.
Hugo Lavenant and Filippo Santambrogio. The flow map of the Fokker-Planck equation does not provide optimal transport. Applied Mathematics Letters, 2022.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Tom. ManiGAN: Text-guided image manipulation. In CVPR, 2020a.
Bowen Li, Xiaojuan Qi, Philip HS Torr, and Thomas Lukasiewicz. Image-to-image translation with text guidance. arXiv preprint, EXXi:2202..02233 00235, 2020b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Shuang Ma, Jianlong Fu. Chang Wen Chen, and Tao Mei. Da-gan: Instance-level image translation by deep attention generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5657-5666, 2018
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instagan: Instance-aware image-to-image translation.
arXiv preprint arXiv: 1088, 2018
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint, arXiv:21 10740, 2021.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text- driven manipulation of StyleGAN imagery. In ICCV, 2021.
12

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scout Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint, arXivv2204.06115, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint, arXiv:2208. 12242, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint, arXiv:2205.1 1487, 2022b.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint, arXiv:21 11. 11.021111122 14, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou Interpreting the latent space of GANs for semantic face editing. In CVPR, 2020.
Jing Shi, Ning Xu, Trung Bui, Franck Demoncourt, Zheng Wen, and Chenliang Xu. A benchmark and baseline for language-driven image editing. In ACCV, 2020.
Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni. GAN-control: Explicitly controllable GANs. In ICCV, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Jianan Wang, Guansong Lu, Hang Xu, Zhenguo Li, Chunjing Xu, and Yanwei Fu. ManiTrans: Entity-level text-guided image manipulation via token-wise semantic alignment and generation. In CVPR, 2022a.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang. Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to mo-ge-to-mmage translation. arXiv preprint, arXiv:2205.12952. 2022b.
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity GAN inversion for image attribute editing. In CVPR, 2022c.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, 2018.
R. Zhang, P. Isola. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain GAN inversion for real image editing. In ECCV, 2020.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In CVPR, 2017.
13

---

Published as a conference paper at ICLR 2023
DIFFEDIT: Diffusion-based semantic image editing with mask generation Supplementary Material
In this supplementary material we provide more details on the experiments and methods presented in the main paper. In Section A provide additional experimental results, including assessment of the impact of the strength of classifier guidance, the impact of using reference texts describing the input image, illustration of the effect of the encoding ratio, more qualitative editing examples, as well as a number of example images and associated reference and query texts on COCO. In Section B we provide proofs that support Proposition in the main paper.
A ADDITIONAL EXPERIMENTAL RESULTS
A.1 ANALYSIS OF NOISE USED COMPUTE THE MASK
In step one our method an editing mask is in- ferred by contrasting noise estimations on noised version of the input image, see Sec-
0.8
Enc. ratio 0.7
35
Enc. ratio 0.8
tion 3.2. In this section, we study the impact of the level of noise added to the input image, by varying its value between 0.1 and 0.8, where 0 corresponds to using the initial image in-
CSFID
30
0.1
0.7
put, and to replacing the imput image with random Gaussian noise. We evaluate the ob- tained operating points on ImageNet with the CSFID and LPIPS metrics when using a encod- ing ratios of 0.7 and 0.8 for DDIM encoding and masked-guide guided denoising in steps two and three of DIFFEDIT. From the results in Fig- ure 12, we find that best results are obtained for moderate values of noise addition of 0.6 and be- low. Indeed, with too much noise added to the input image, it is difficult to correctly identify visual elements in the input image. We use value of 0.5 all our experiments.
25
0.4
0.6
26
28
30
32
LPIPS
Figure 12: Impact of the noise added to input im- age when computing the mask, for a encoding ra- tios of 0.7 0.8 on ImageNet.
A.2 CLASSIFIER- GUIDANCE
In diffusion models, the noise estimator eg can be conditioned on text describing the image, which provides signal to guide the noise es- timation process (Nichol et al., 2021; Saharia al., 2022b). Ho Salimans (2022) intro- duced classifier-free guidance, a technique to greatly improve generation quality and image- text alignment in text-conditional diffusion models. It consists in training both a condi- tional and unconditional model by dropping the conditioning text at train time with fixed prob- ability, e.g. 10%. Then, after training, at each step & during decoding, the noise estimation (x1, Q,4) is extrapolated by using the uncon- ditional noise estimation ea (x,, 0, !) as origin. Formally, the noise that used is
90
Ours (Guidance 5) Guidance
80
Guidance
70
Guidance
CSFID
60
Guidance
50
40
30
10
15
20
25
30
35
LPIPS
Figure 13: Ablation on the value of the classifier- free guidance parameter. On ImageNet, we find that a value at least must be used to get good results. We chose to use 5, which is the default value recommended for generation.
c= cx 0,t) + Xxco(x,, Q, (x, 0,))),
where À is the classifier-fres guidance parame- ter. We study the influence of this parameter DIFFEDIT in Figure 13, finding that similarly to generation, a value above 3 yields the best results. Without classifier-free guidance, the obtained trade-off is competitive. For experiments we default guidance value of À
14

---

Published as a conference paper at ICLR 2023
A.3 EXPERIMENTS ON COCO FILTERING
Here, we investigate why there is little difference between using or not the reference text compute the mask on our COCO
90%
90%
30,0
w/ ref
80%
31.2
DiffEdit
w/ ref text
80%
DiflEa
70%
31.0
DifEdit
29.5
ref.
ref text
CLiPSeore
CLiPS
30.8
queries. In Figure 15 we show several editing queries on the COCO dataset taken from the BISON dataset (Hu et al., 2019). Generally, the text query describes a scene similar the one in the input image, and it possible match the text query by editing only a fraction of the input image. However, we find that while queries have been built to be close to caption of the input image, most of the time the query is not well aligned with the caption. We create a filtered version of this dataset, for which queries are structurally similar to the caption, i.e. where only a few words are changed, but the grammatical structure stays the same. We use the filtering criterion that the total number of words inserted/deletccd/ppllaes must not exceed 25% of the total number of words in the original caption, resulting in a total of 272 queries out of 50k original queries. In Figure 14 we compare results with and without filtering, and observe that for the images with small caption edits the gain of DIFFEDIT (w/ ref. text) compared to Encode Decode is somewhat larger than on the unfiltered dataset. Moreover, using the original caption reference text compute the mask gives higher CLIPScore, especially at high encoding ratio. This illustrates that a well chosen reference text helps to generate better editing masks.
29.0
60%
70%
28.5
30.4
60%
500
28.0
30.2
50%
27.5
40%
30.0
10
15
20
25
10
15
20
LPIPS
LPIPS
Figure 14: Results on COCO: unfiltered (left) and filtered (right). While having a small impact overall, for the filtered set using the ref- erence text beneficial, especially high encoding ratios, e.g. 90%.
Dataset
COCO
COCO
COCO
Input Image
modern style bathroom with a large tub and shower and tile floor.
Caption
A large tall tower with clock on top.
Three giraffe stand near fence as two women watch them.
Text Query
bathroom with sheer curtains framing the tub.
A tall clock tower with trees all around.
Two giraffe standing next to each other near brick building.
Figure 15: Editing queries on the COCO dataset.
A.4 VISUALISATION OF THE IMPACT OF ENCODING RATIO
We show visual results for ablations of our two main components, mask inference and DDIM en- coding, in Figure 16. The resulting methods are SDEdit, -ecode- DIFFEDIT w/o Encoding, and DIFFEDIT. We demonstrate the qualitative behavior of these different methods, varying coding ratios between 30% and 80%. Compared to SDEdit, Encode-Decode allows to better match the query with less modifications of the main object and the background, especially at 60% 70%.
15

---

Published as a conference paper at ICLR 2023
Mask inference allows maintain exactly the background. Using DDIM inference top of mask- based decoding allows to better retain of the content inside the mask, especially at 70% and 80%, c.f. row 3 vs. 4.
Method Name
Encoding Mask
30%
40%
50%
60%
70%
80%
Input Image
SDEdit
X
X
Query
Bald Eagle
Encode- Decode
X
Musk
DiflEdit wio Enceding
generuted DiflLdit
DiffEdit
Figure 16: Qualitative ablations of the mask and encoding components, using different encoding ratios from 30% 80%.
A.S ADDITIONAL VISUALIZATIONS AND QUALITATIVE RESULTS
Figure 17 illustrates editing examples on Imagen, in comparison with other mask-free editing meth- ods. DIFFEDIT generally performs more targeted and accurate edits, leaving more of the original image in tact where possible. Consider for example the first column of Figure 17, where DIFFEDIT leaves the guitar as it, while other methods make unnecessary and unrealistic changes to the guitar.
Additional qualitative examples COCO images are shown in Figure 19.
Figure 20 shows several failure cases of semantic image editing with DIFFEDIT. Some failure modes are inherited from the generative model itself: models trained web-scrapped image-text data are known to struggle with understanding spatial positions in images, spatial reasoning. and counting (Ramesh et al., 2021). Others specific insert objects, because the mask often seeks an "anchor" visual element to insert an object, see first column.
mask-based method, like the difficulty to
A.6 DETAILS ON COMPARISONS WITH OTHER METHODS
On the COCO and Imagen datasets, we do not compare with ILVR, since it cannot be used within the latent diffusion framework: the method needs image downsampling and upsampling, which does not work well with the latent spaces used in latent diffusion. Even adapted with diffusion model without latent spaces like Imagen, we do not expect the CLIP-LPIPS trade-off to be favorable for this method, given the high editing distance obtained Attention Control Hertz et al. (2022), recent method for text-driven image editing based on the unreleased Imagen diffusion model. The method is very recent and has been adapted to use with Sta- ble Diffusion at https: ///::////iii ///gttttu///// com/ holm/ble oc97 wc97/Crossaa //Crossstteettt iestentiiinnn LonControll....ooooooooo performed lightweight hyperparameter search to optimize the CLIP-LPIPS trade-off on subset of Imagen images. We generally find that this re-implementation. while producing edited images structurally similar to the input, changes local features more than SDEdit, leading to generally high LPIPS distances, resulting in CLIP-LPIPS trade-off not competitive with other methods on our COCO and Imagen benchmark. In particular, LPIPS distance are high on the COCO dataset where text query and reference text have high edit distance on average, whereas Cross-Attention Control was designed to perform well for most exactly match. Given that our results are based the unofficial re- implementation, that they are temporary and we will update them when the official code (or official adaptation for Stable Diffusion) released.
ImageNet. Instead, compare against Cross-
We have
editing, i.e. the input and target text should al- caution
16

---

Published as a conference paper at ICLR 2023
Inpul
Text Query
glasses cowbuy
glasses cowbuy hat
riding bike skateboarding
british shorthair cal -t fuzzy panda
fuzzy punda racooon
FlexIT (res 256)
Cross- Attention Control
SDEdit
DiffFdit
Figure 17: Example edits from Imagen, comparison with other mask-free editing methods.
17

---

Published as a conference paper at ICLR 2023
Input Image
Editing query
Pancake with burger
Bread - burger
Truck old truk
Keyboard Piano keyboad
Mashed potatnes eri
SDEdit
Encode- Decode
DifEdii Mask
DiffEdit
w/o Encode
DiftEdit
More qualitative examples on COCO. Baseline methods are shown for comparison. The mask is sometimes bigger smaller that one could expect: In column 3. it is larger, but there are few edits outside the requested bread + burger transformation (except for the wine bottle label), which is not the case without DDIM encoding. In column 4, the mask does not cover the interior of the truck, but this does not affect the edit quality.
Figure 18:
18

---

Published as a conference paper at ICLR 2023
Input Image
Editing query
playing frichees - A dog playing with teskethalll balls
dog plainng with frahhos A woll playing fiisbess
House and cartiage Ilouse and moce
House carragge middlo-age etalle and cariigge
A motorbike and baleasy -n mosorbike planes balcony.
motorbike A house and vintage old
SDEdit
IALL
Encode- Decode
DiffEdit Mask
DiffEdit w/o Encode
Diffdit
Figure 19: More qualitative examples COCO. In the first column, the color of the objects to be edited maintained, which would not be the case with regular inpainting methods. Contrasting similar text query and reference text allows select the object to be edited.
19

---

Published as a conference paper at ICLR 2023
Input Image
crssty
gey and yill lead A maddon lutkked unala pame with yallow hand
Editing Query
right
Thon gloffiie on
deg
the cifld lawce
hov bellon
masson
bhat cir on the oegtor sannge garage lest
white will on the night
DitEdit Mask
DitEdit Output
Failure
Onjer louree at wring place
Object inurnddur sine plase deyect
Object lesent
Madk decr bullde ebjoct be odted, both sijeet
Meek dees and buclans oisec su be odkidd
pesiion
Figure 20: Illustration of failure modes. In the first two columns show difficulty to insert an object smooth region of the image. In column three the mask fails identify region where to add the zebra. Columns 4 and 5 show mask identification errors, where multiple similar objects are included in the mask whereas matching the text query only requires to edit single object. both cases this results in over editing. Col. shows the failure to change spatial relation in the image.
20

---

Published as a conference paper at ICLR 2023
B THEORETICAL RESULTS
Here, we prove the bounds given in the main paper. We reused notations from Proposition in the main paper. We also discuss links to optimal transport.
B.1 PROOF OF SDEDIT BOUND
Proposition 2. Suppose that ||ea Q,t)ll < for all E X, t [0,1]. Then
Q)II Xx (x(G,(xo +), - (C + 1)r
(6)
(xppQ)=<p e~,1(
Proof. Let X,, Gx0 E) .r x, and yo Dy Q). Then
Xr oo
yoll = -r
yo yn || = c0( Q, ()drl & Cr. Vao
(7)
Vor
Since Xr var XD + TC, we have |xo - yoll S ||xo + re yoll ||T||| & Cr + 7 which concludes
the proof.
In the SDEdit paper (Meng et al., 2021), a proof similar to what we state is given, with three main differences: (i) the proof is given in the case of variance-exploding Stochastic Differential Equation (VE-SDE), which needs adaption for our setting which uses variance- variance-preserring SDE; (ii) the bound derived the case of a stochastic differential equation. whereas we use deterministic DDIM process; (iii) the bound is given by controlling the probability tail, whereas we only consider the expectancy of edit distance. However, despite these differences, the spirit of the proof is the same as here.
B.2 PROOF OF PROPOSITION 2
Proposition 3. Suppose that c0(-, Q,t) t) is Kipiishhhitz and K2 defined as
K2 x2(xx) max (Ex(()))))) Q, co (Ec 0, t)
(8)
tE|0.1]
Let K₂ Ex (x00). Then for all encoding ratio r, with t = v or -1-1
K2T VF2 +1
T 72 + Ki
Ex, |xo Q))
<
(9)
Proof. Let a be a time dependent variable defined as (t)) = Va;'- Let = =
xV1 + and v = =yv1+ +/1+00 02 u and v are solutions of the following differential system:
du/: = co / 1 2,00 d)o
(10)
dv/: = co 1 02, 1,t)do u(r) v(r) Ex,) 1
(11)
(12)
Let ||u v/I. then w/.
0 and
dw|: < |/du/ - dv/all [x(e,(x, (x, 0,1) - en(y, 1o
(13)
(1(e(( 0,t) t0 (x, Q, f)lldo + +l((e((x, Q,t) ealy, Q, dll
(14)
mz (xo + K1 / x y yl
(15)
K1
(x()
do.
(16)
v1 + 0
21

---

Published as a conference paper at ICLR 2023
By integration get
(t) S (x0) - + f."
Ki VT+ w(o)da.
vT+ozu(0)do.
From here we can apply Grönwall's inequality:
K1 (Lo vI + st (ds)
w() S (x0) exp
(17)
s (x0) exp
(Ki (KllogT++ + VT2+
(18)
s xx( (+ +2)+1)
(19)
Which finally gives
|xo - yol S (xo(T ( + v Ki
(20)
+1
Taking the expectation w.r.t. the input image X, gives the final result:
Ex, 1 (r((E(xo)) Q)
V (T
K2T
V+2+1) K1
(21)
which concludes the proof.
B.3 LINKS TO OPTIMAL TRANSPORT THEORY
The reverse DDIM encoder E, maps the distribution of images Po PD to the distribution p,- of images noised at timestep r. Khrulkov & Oseledets (2022) suggested that E, could be an optimal transport map between Po and pr, minimizing the transport cost Ex, ||x, E, (xo) I13. This means that the encoded images are, on average, as close as possible to the input images, while following the correct distribution pr- would entail that the unconditional decoder D, E, would be and Po. and moreover that the conditional decoder D,(,, Q)
an optimal transport map between would be an optimal transport map between the distributions pr((IQ) and po(-IQ) conditioned by text description Q. Under the hypothesis that p, is very close to pr P-(-1(), then the Encode-Decode algorithm would the combination of two optimal transport maps F,, and Dr(: Q)) mapping po to Pr and then pr = P-(-1)) to Po(-IQ). This is very interesting property and we make the connection with the desired properties of semantic image editing, which can be expressed as an optimal transport problem. Given two distribution of images Pt, P2 (lets say cats and dogs), the aim to find the function f that performs the expected edit (changing images of cats into images of dogs) while minimally editing the image, which can be expressed mathematically as:
arg min Ex|[x - f(x)]] s.t. f
P2 f
(22)
where f+ is the fush-forward measure. The function D, Q) E, not a solution of this optimal transport problem, because (i) it was proven that the reverse DDIM encoder is not the optimal transport map for some distributions (Lavenant & Santambrogio, 2022), and (ii) the composition of two optimal transport maps is not necessarily an optimal transport map. However, experiments and numerical simulations suggest that E, is very close from an optimal transport map. would be interesting to study the "optimality defect" of E, and of the editing function D,(, Q) E,. We leave this for future work.
22


Reviews:
[{"review_id": "tMxYoRb2-YQ", "reviewer": "Reviewer_hxgu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DIFFEDIT, a new method for image editing based on diffusion models. The main contribution is a method to automatically discover the areas that need to be edited by contrasting the predictions conditioned on different texts. Besides, the paper shows that using the latent inference capability of DDIM achieves more consistent editing. Experiments on ImageNet, COCO, and an Imagen generated dataset demonstrate that DIFFEDIT achieves better image editing than previous baselines.", "summary_of_the_review": "While I feel the proposed method is pretty simple, it is reasonable and effective in producing region-specific and consistent image editing as proved in experiments. Object specific image editing is an important problem and I think this work would be quite useful. A minor weakness in qualitative evaluation could be further improved. To summarise, I tend to accept this paper.", "strength_and_weaknesses": "Strength:\n\n- The paper proposes a new method to automatically estimate the mask that needs to be edited, which reduces users' effort. The method of contrasting the predictions conditioned on different texts looks reasonable to me.\n- DIFFEDIT proposes to use the reverse DDIM step to encode the image, which is shown to lead to more consistent image editing results.\n- Combining the proposed two techniques, the method achieves better image editing than previous methods both qualitatively and quantitatively on several datasets.\n\nWeaknesses:\n\n- For the COCO dataset, only qualitative results of the proposed method are shown. It is recommended to add a qualitative comparison with other baselines in the Appendix.\n- For the current method, the predicted mask is much more reasonable when a reference text is provided. However, in the application of real image editing, the reference text is often not available, so a minor issue is that the author needs to provide both query text and reference text for the method to perform better. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i0rXmievZl", "reviewer": "Reviewer_zQQb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a 3 steps image editing pipeline that 1) generate segmentation mask from the change in text query, 2) encode the image with diffusion process until a time step ‘r’, and 3) decode it back to the image via reverse diffusion process condition on the text query, and with mask as the guidance. They show results on ImageNet and coco dataset, as well as image generated from imagen. ", "summary_of_the_review": "Unfortunately, I don’t think I currently see enough contributions to recommend acceptance. Given the weakness I mentioned, I think it is slightly below the bar for acceptance. ", "strength_and_weaknesses": "Strength\n - theoretical analysis is nice to have\n - the generated mask seems reasonable\n - the method is simple. Auto generate the mask from 2 version of slightly different texts then encode/decode to the appropriate image.\n - experiments are on (relatively) large scale datasets \n\nWeakness\n - The use case is not quite clear to me. The mask generation from the different in text query and reference text means that you need to have both. So from what I understand, this is only applicable for image editing on an existing text-to-image pipeline? (Also Isn’t that more like giving more control over image generation than an image editing?)\n - I think the simplicity of the method is both the strength and the weakness. It is good that it is simple, so more people can use it. But it also mean there isn’t much insight/impact gain from reading the paper.\n- From the result figure, it is not an obvious improvement to me among each baselines. Using mask does help with keeping the background the same as the original query, but that seems to be the only benefit of the proposed method. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_mSgItV9dpP", "reviewer": "Reviewer_WyT5", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel method for text based image editing, that addresses the shortcomings from previous methods:\n- either a mask had to be provided by the user\n- or the edits modified the background\n\nIn this paper the user does not have to provide a mask and the background will not change either. They achieve that by estimating the mask instead of requiring the user to provide it.\n", "summary_of_the_review": "The paper have strengths in many areas. Unless I missed something, it is a clear accept.\n", "strength_and_weaknesses": "Strengths:\n\nThe process is extremely simple: \n- step1: estimate the mask (that a user otherwise would provide)\n- step2: run DDIM encoding\n- step3: run DDIM decoding conditioned on query text, and keep pixel values outside the mask\n\nThe key idea is very intuitive:\n\"When the denoising an image, a text-conditioned diffusion model will yield different noise estimates given different text conditionings. We can consider where the estimates are different, which gives information about what image regions are concerned by the change in conditioning text.\"\n\nThe paper is very well written.\n\nThe solution is supported by theoretical analysis, where they show bounds on how far the edited images are.\n\nAblations show the main effect on design choices (fig 5, fig 6) and quantitativelly on the main hyperparameters (fig 6 binarisation threshold and fig 4 encoding ratio).\n\nThe method is extensively evaluated on different datasets and against other methods, and it beats the SOTA (fig 4)\n\nWeaknesses:\n\nI did not find any", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7qsHItervpR", "reviewer": "Reviewer_ZwKd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for performing text-based semantic image editing. The key ideas are \n1) identifying where to edit by comparing the image differences between query text-guided and reference text-guided (or unguided) image generation and\n2) use DDIM encoding for preserving the contents that are outside the generated masks. \n\nFor 1), compared to the existing semantic editing method, the proposed mask generation automates the step of manually specifying the mask for editing. \nFor 2), the DDIM encoding preserves the background content more faithfully when compared with current practices (that add noise to the original image as in SDEdit).\n\nThe method can achieve semantic editing while maintaining the background contents in the original image. \nThe paper evaluates the method in three settings, ImageNet, images generated from Imagen, and COCO. The experimental results validate the capability of the proposed method for image editing.", "summary_of_the_review": "The paper combines the ideas of spatial masking and DDIM encoding to improve the quality of semantic image editing. None of the techniques are new, but it appears to be an interesting combination in this application context. The paper shows strong evaluation results on three datasets and reports convincing quantitative and visual results. I think the community would benefit from this simple approach for semantic image editing. ", "strength_and_weaknesses": "Strength:\n+ The paper is very well-written. The exposition is clear. The figures are informative. \n+ The proposed approach is technically sound. \n+ The evaluation of three datasets (ImageNet, images generated from Imagen, and COCO) is solid and convincing. Both the quantitative and qualitative results demonstrate the effectiveness of the method.\n+ The ethics statement and potential misuse of the techniques are thoroughly discussed.\n\nWeakness:\n- The concepts of predicting spatial masks for semantic image editing have been extensively explored in other image-to-image translation literature. Examples include:\n[CVPR 2018] Da-gan: Instance-level image translation by deep attention generative adversarial networks\n[NeurIPS 2018] Unsupervised Attention-guided Image-to-Image Translation\n[ICLR 2019] InstaGAN: Instance-aware Image-to-Image Translation\nThe difference here is that the generative models at the time were GANs instead of diffusion models.\n\n- Another core difference is the use of DDIM encoding as opposed to directly adding noises in the original image. DiffusionCLIP [CVPR 2021] also uses the DDIM step to encode the input image. The key difference is that the proposed method relies on a pre-trained text-2-image diffusion model as opposed to CLIP used in DiffusionCLIP. This allows us to avoid the costly model fine-tuning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%