MDP: A Generalized Framework for Text-Guided Image Editing by Manipulating the Diffusion Path
Authors: Qian Wang
Biao Zhang
Michael Birsak
Peter Wonka
Venue: Accepted by TMLR ()

Abstract: Image generation using diffusion can be controlled in multiple ways. In this paper, we systematically analyze the equations of modern generative diffusion networks to propose a framework, called MDP, that explains the design space of suitable manipulations. We identify 5 different manipulations, including intermediate latent, conditional embedding, cross attention maps, guidance, and predicted noise. We analyze the corresponding parameters of these manipulations and the manipulation schedule. We show that some previous editing methods fit nicely into our framework. Particularly, we identified one specific configuration as a new type of control by manipulating the predicted noise, which can perform higher-quality edits than previous work for a variety of local and global edits.

Qian Wang Published in Transactions on Machine Learning Research (09/2024)
MDP: A Generalized Framework for Text-Guided Image Editing by Manipulating the Diffusion Path
gian.uang@kkousteduu edw.
King Abdullah University of Science and Technology
Biao Zhang
biao. bioo.zhangg@keuustccu..ek
King Abdullah University of Science and Technology
Michael Birsak
bhichael.breakkkaake
King Abdullah University of Science and Technology
Peter Wonka
weteer.onkookaaust.com edu.
King Abdullah University of Science and Technology
Reviewed on OpenReview: https: openrevieww net/ forum? id- 2666
Abstract
Image generation using diffusson can be controlled multiple ways. this paper, we systematically analyze the equations of modern generative diffuusion networks propose framework, called MDP, that explains the design space of suitable manipulations. We identify 5 different manipulations, including intermediate latent, conditional embedding, cross attention maps, guidance, and predicted noise. We analyze the corresponding param- eters of these manipulations and the manipulation schedule. We show that some previous editing methods fit nicely into our framework. Particularly, we identified one specific con- figuration as a new lype of control by manipulating the predicted noise, which can perform higher-quality edits than previous work for variety of local and global edits.
1 Introduction
"Cat" "Dog"
"Red" "Purple"
Remove "Stuf""
Mix "Teapot" and "Shoes"
Local Edits
"Table" "Galaxy"
"Fall" Winter"
"Painting" "Children's drswing"
Stylise "Blanket"
Global Edits
Figure Our proposed manipulation of predicted noise, which fits into our proposed editing framework MDP, can do multiple local and global edits without training finetuning.
Previously, image editing using GANs has achieved great success (Isola et al. 2017 Zhu et al. 2017 Choi et al.) 2018 Wang et al. 2018 Huang et al.) 2018 Kim et al. 2017 Bau et al,
2020 As large-scale
text-to-image datasets bocame available, text-guided image synthesis and editing has obtained increasing
attention (Ramesh al. 2021 Crowson al. 2022 Ding al. 22021] 2022 Gafni et al. 2022 Generative
diffusion models Ho et al. (2000 Rombach al. (002 Saharia al. (0022): Nichol et. al. 2022 Ramesh

---

Published in Transactions on Machine Learning Research (09/2024)
et a.l
2022
are also powerful tool for multiple image processing tasks, such inpainting Lugmayr
et
al.
(2022)
Choi et al. et al. (2023), text-guided image editing Brooks et al. (2022); Kawar et al. (2022): Hertz et al.
(2021); ; Li et al. (2022); Xie et al. (2022).
style transfer Kwon & Ye (2022)
Bansal
[2022);
; Tumanyan et al. (2022). map-to-image translation Voynov et al. (2022) Avrahami et al. (022a and
segmentation Burgert et al. (2022); Baranchuk et al. (2021).
We are interested in simple and effective image editing methods that do not require retraining, fine-tuning, training of auxiliary network. Specifically, for text-guided image editing, we can leverage pre-trained diffusion models to perform editing tasks real images. An arbitrary real image first be embedded into
the diffusion latent space to obtain an initial noise tensor (Wallace et al. 2022 Mokady et al.] (022). Then, by manipulating the diffusion generation path starting from this initial noise tensor, we can edit the input image.
50
40
30
20
10
5
3
2
Figure 2: Assume the total number of sampling steps T= 50, The number on top of each image shows the number of diffusion generation steps for which we use c(A) " "Phto of a cat sitting next to mirror" invert and then denoise. For the remaining diffuusion generation steps, we switch to (B " "Pholo of a liger silling next to a mirror" to denoise. If we change condition in the early stage (e.g., 30 and 20), can preserve the layout of c(4) while including the semantics from cBB), If we switch at later stage (e.g., 5, 3, 2, 1), the layout of (A is hardly maintained in the result.
Given an imput image, perform edits guided by a condition (for example text prompt class label) while keeping the overall layout from the imput image. We can see this editing task as combining the layout from the input image with new semantics from the condition. Since diffusion models generate images progressively, the layout of an image is generated during the early timesteps of the denoising process, while
the semantics, iee. the texture, color, details, are generated during later steps (Hoogeboom et al. 2023 Liew et al. 2022). Therefore, is intuitive that each timestep should be controlled separately.
For each denoising step, there are several components that can modify during the diffusion process. Intuitively, we want to denoise using information from the input image, for which we want to preserve the layout, in the early stages, and then the components of the new condition in the later stages. At this point, two questions arise: (1) Which component of the generation process should be modified? (2) For which denoising steps should the modification take place?
In order to tackle the problems, we analyze the equations of modern generative diffusion norks to find out which variables could be manipulated and identified different manipulations that are suitable: intermediate latent, conditional embedding, cross attention maps, guidance and predicted noise. We take the manipulation of attention maps from the seminal paper Prompt-to-Prompl (P2P) Hertz et al. (2022) As result of the analysis, we introduce a generalized editing framework, MDP (Manipulating the Diffusion Path), that contains these 5 different manipulations, their corresponding parameters, and a manipulation schedule for them. Some of the manipulations, though simple and have not been explored before, are proved to be meaningful for different editing applications.
We categorize the image editing applications into local editing and global editing. We find that different manipulations differ from the degree of preserving the original layout. If too much original layout is preserved, new semantics has difficulty injecting into the edited image; while if too less original layout is preserved, original layout has to be injected for more diffusion sampling steps, while original semantics inevitably leaks into the edited image in the later diffusion sampling steps.
In this design space, we identified one particular configuration, MDP-G, that yields very good results for a variety of local and global image editing problems shown in Figure ! The results from MDP-c, are more consistent in quality than previous work (P2P) and other possible configuratioss. That means, MDP-c, has
2

---

Published in Transactions on Machine Learning Research (09/2024)
the best degree of layout preservation compared to other manipnlations. We advocate for the use of MDP-c: as new method for diffusion-based image editing. Our work makes the following contributions:
We present framework for generalized design space to manipulate the diffusion path that includes multiple existing methods.
We analyze the design space to find good configurations suitable for editing applications.
We propose new solution by manipulating predieted noise that can solve practical local and global editing problems better than other configurations and previous work.
2
Related Work
Image diffusion models. In recent years, the interest in deep generative models has gained strong mo- mentum and many different methods have been proposed to create high-quality samples from a magnitude of different data domains. One kind of architecture that currently stands out are diffusion models (Ho al.,
2020 Dhariwal & Nichol, 2021 Karras et al. 2022
In order to obtain better control over the denoising process and the generated content, several works pro- posed text-conditioned image synthesis using diffusion models and cither CLIP guidance or classificr-free
guidance (Nichol el. al. 2022 Ramesh et al. 2022 Saharia et al. 2022).
Several approaches focus on different optimization strategies and either propose to generalize DDPMs via a
class of mon-Markovian diffusion processes trade off computation for sample quality (Song et al. 2021 or
to do the diffusion process in the latent space of a pretrained autoencoder (Rombach et al.) 2022 instead of the RGB pixel space.
Image editing with diffusion models. InstructPix2Pix Brooks et al. (2022) can edit images by following
user instructions. Paint by Example Yang et al. (2022) allows users to replace an object with a conditional
example image. ControlNet Zhang Agrawala (2023) trains a task-specific condition network on top of a pre-trained diffusion model and supports various edits based on a specific training set. All these works usually require re-designing of the model, collecting training data and a long training time to do image translation.
Blended DIffusion Avrahami et al. (202bb) and Blended Latent Diffusion Avrahami el al. 2023 adopt a user mask as input and design specific loss function to obtain good text-guided image editing results within the mask rogion. UniTune Kawar et al. (2022) and Imagic Kawar et al. (002)) can do realistic text-guided image editing by either just linetuning the diffusion models or the models and text embeddings together. However, because of the optimization process, it takes several minutes or more to generate single image.
Recently, there many interesting works that only utilize pretrained diffusion model to do text-guided image editing without any training or finetuning (Meng et al. 2021 Radford et al. 2021 Couairon et al.
2022 Liew et al.] 2022 Hertz et al. 2022 Park et al.] 2022 Parmar et al.) 2023 Tumanyan et al. 2022 DiffEddit Cousiron et al. (2022) automatically generates a mask by computing the differences between the noisy latent generated from an imput image and a conditional text prompt. However, because of its inpaint- ing nature, DiffEdit cannot perform global editing operations like converting a photo to an oil painting. Liew
MagicMix Liew et al. (2022) interpolates the noisy input latent and the denoised latent mix the objects that have large semantie differences. Prompt-to-Prompt Hertz al. (2022) proposes to manipulate the cross-attention maps corresponding to the changes between the input and guided text prompt. Concurrent work pix2pix-zero Parmar et al. (2023) computes an editing direction to edit an image based on a provided prompt, combined with a cros-aattention mechanism to preserve the layout of the input image.
Compared to prior works, we propose a generalization that includes multiple previous approaches as special case. We also highlight a novel manipulation, that suitable better perform wide range of local and global edits than previous methods. Our edits do not require masks for an input image. Our method is purely based a pre-trained diffusion model and does not require finetuning.
3

---

Published in Transactions on Machine Learning Research (09/2024)
3 Design Space for Manipulating the Diffusion Path
We provide the details of sampling using diffusion model, image editing using diffusion model and inversion for diffusion model in the Supplementary Materials. Here introduce the design space of our framework. We abstract our framework to linear interpolation operations and no higher-order operations are explored in this work.
Table 1: We use the superscript (A) to represent outputs obtained from condition c(A) and (B) for condition c( The edited outputs are represented with superscript (+), is number between 0 and 1. 8 is usually positive real number.
Changes
Excist. Methods
Inter. Denoised Interp.
Al
GemPPath (x0,,e(4),
(1 woss)
<7" co (x1", ee ,.,
Liew 2122
Inter. Denoised Masking
o M)
x/11
-DDM (xlt))
Coustton
20222
(x _eft),))
Condition Emb. Inserp.
Gen(Xy, el4).()
elt) - we(+) w)
x DDIM (x)*)) (.)..)
Manip.
(A)
Gen(xy,
M (er(x((") ,, (10)))00),
(x11)
DDIM (x!"). (4).
Hertz at nl.
2022
Chefer et al 2023
c (x1) cl4),e) .c)
Guidunce
py-~ ~ N)O.1)
of = to (c) .e!m
[1 DDIM (%)))
To & Salimans 2022
== (1 5).
Pred. Noise Interp.
((055)) (21))))))))))))))) - GenPath (x0),e))..)
we!") w)de
DDIM (x1%.4%))
Pred. Noise Masking
Mod" M.
Simply switching to another condition during denoising is a very straightforward way to add new semantics to the input image when doing text-guided image editing. We perform a simple experiment in Fig. 2 to illustrate that the early timesteps in the diffusson process contribute to the layout, while the later timesteps are adding semantics and details to the image. Our goal is to systematically analyze the design space for manipulating the diffusion path over time to perform these types of edits. As result of our analysis, we propose MDP generalized framework for text-guided image editing and show how concurrent and previous methods fit our framework. A summary can be found in Tab.
All manipulations in our framework are time-dependent and have parameters, e.g. interpolation parameters. MDP allows the user to specify different types of manipulations and different manipulation parameters at each timestep, which we call the manipulation schedule.
(4) as input, along with a condition c(4) that is used to generate x! and a new Assume we have an image x condition (B)), We first invert the image x manipulations:
(A) to get an initial noise xT. We identify the following types of
Intermediate denoised output. Starting from XT, we can generate image X9 using the new condi- tion c3 Assume we have all the intermediate latents for the generation of x((1) { (
(B)
GenPath (xr,c(4) .). We denote the intermediate latents in the new path as x(+). We can modify the intermodiate outputs x, (4) and x(*)) with either lincar interpolation or masking. We can control the resulting image by changing the interpolation factor w (or the binary mask M).
(+) = x
(4) (1) w)x(")
(1)
or
x; M θ x
(4)
+ (1 M) O n
(2)
When we start this manipulation, DiffFdiit Couairon et al. (2022).
x
(+)
x
(B)
The strategy is used in MagicMix Liew et al. (0022) and
4

---

Published in Transactions on Machine Learning Research (09/2024)
Stpp T
Step
A0
elift
DDIM
DDIM
DDIM
DDIM
J.
(A)
A0
Stpp
Step from
Step Avei
Step
LTI
ODIM
DDIM
DDIM
DDIM
Stopp T
Step fusss
Step fask
Step
DDIM
DDIM
DDIM
DDIM
Figure 3: Predicted Noise Manipulation. The top branch is inverted from a real image given condition c(4) Photo of rabbit on the grass". The bottom branch is generated using condition ( "Photo of rabbit in a library". We copy the predicted noise from step bmaz to Lmin the top brauch, then use (B) to denoise and generate the middle images.
Condition embeddings.
Given two conditions (A) and c( we can change (A)) timestep t,
cf = (c( + (B)
(3)
where cf will be used as the new condition that embeds the information from both condition e(A) and c(B),
Cross attention. Recent condition diffusion models inject condition information by cross altending between and image features. Prompt-to-Prompy Hertz et al. (2022) and Attend-and-Fxoite Chefer et al. (2023) find that modifying the cross attention maps can give interesting image editing effects. Here, we do not dig into details of how to manipulate the aross-attention maps. Instead, we simply write the manipulation in functional form M,
( - ((0 (x(A)),
c(A) .o(8))
(4)
Guidance. Classificr-froe guidance Ho & Salimans 2002 proposod
( =co (xL, c()),) + B (co x,, c(a), 1)
cg (xe, Ø :))
(5)
where 3 is real number and Ø denotes an empty condition. The 3 is often called "guidance scale". The Similar idcas are also used in Bansal et al. This can be generalized to
term 6o (x,, Ø.t) can be seen as unconditional output of ca. h2223: Liu el al. (2023): Brack et al. (2022) for image editing.
=69 Xx,, c(a), ()) + 3 (Eo Xxx, c(4),1) en (xc, (..,,11)))
(6)
5

---

Our manipulations and baselines. Given real input image (() we first use Null-text Inver- sion Mokady et al. (2022) together with text-condition c(A) to obtain an initial noise tensor xy- The initial condition c can be either an empty text, a manual prompt provided by the user or caption generated by image captioning tools. Another given condition (!) that is used to guide the editing process would induce the generation of output image x starting from the same initial noise tensor Xy. As do Published in Transactions on Machine Learning Research (09/2024)
Intuitively, with equation 14 want the output to have more characteristics from c(4) when generating
using the new condition c to preserve the layout. If 8 is a number between [-1, 0J, we obtain this linear interpolation,
w w.Ee co (x,,c! + (1 w)
w.)..a.. co X,, c (B)),
Predicted noisc. Both the equation 4 and equation 14 are modifying predicted noise ct- Inspired by this, we investigale another manipulation. In timestep I, we interpolate the predicted noise 6 (4) with the predicted noise that using condition c(B), Assume we have all the predicted noises for the generation of x (A).
(xr,c" {xx .6 e((")) [=(T.....)) = GenPath (x, (xr,c()) ,t),
(8)
we can mix the two predicted noises,
(+) = w we xxx,, ,((4) + w) c())) (
(9)
where when we start this manipulation, x, xx
(B) We show the pipeline of this method in Fig.
4 Experiments and Analysis
Table 2: Quantitative results of MDP-E; compared with baseline methods.
MDP-r:
MDP-c
P2P
MDP-3
DiffEdit
MDP-e;
LPIPS4 CLIP scoret
0.3813
0.4111
0.2900
0.4528
0.2348
0.3484
27.5084
28.4560
28.2487
27.9109
27.6915
28.3228
CLIP directional similarity?
0.2262
0.2062
0.1938
0.1935
0.1851
0.2265
4.1 Settings
Editing tasks. We describe small taxonomy for image editing applications that to analyze framework and compare it to previous work. We divide all common image editing operations into two categories: local editing and global editing. We provide the details in the Supplementary Materials.
not use mask an input, we only consider linear operations that do not involve mask:
(4) MDP-2,: intermediate latent interpolation. We linearly interpolate the intermediate latents x]"
and x (" using equation [ This manipulation differs from MagicMix Liew et al. (2022) by using Null-text Inversion to obtain the initial noise tensor Xy rather than directly adding noise.
MDP-c: conditional embedding interpolation. We interpolate the condition c(A) and c by using
equation 3
P2P Crompp-to-Promphe Hertz et al. (0022) cross attention manipulation. P2P manipulates the cross attention maps and self attention maps according to the changes in the new text prompt compared to the input prompt.
MDP-5: guidance. We use condition (A) a guidance to inject the layout while generating new semantics by using equation 15

---

Published in Transactions on Machine Learning Research (09/2024)
MDP-c;: predicted noise interpolation. We mix the predicted noises when generating input image (4) with the noises using condition c(B) by following equation X
DifTEdit Couairon et al. 2022): intermediate latent interpolation. We also adopt DiflEdit as another competitive baseline for more comparisons.
Comparing MDP-r,, MDP-G and P2P from their formulations, MDP-x; directly injects the intermediate latent from the original image, which has the largest degree of preserving the original layout. As for MDP-er which injects the predicted noise and can only partly affect the new intermediate latent; while for P2P, it manipulates the attention maps which can only partly affect the new predicted noise. Therefore, P2P has the weakest degree of preserving the original layout. Due to the space limit, we show the qualitative results of DiflEdit in the Supplementary Materials.
Manipulation schedule. We investigate how the methods work under simple schedule using default settings. We start the manipulation at step maa and end at step bmin- The total number of timesteps of a manipulation is denoted as Twy = tmaz - binn We perform edits only during these TMr steps. For the remaining steps, we simply use c²²) to denoise the noisy latent. We vary tmar and tmin and mamually select the best result for each method. We fix the interpolation factors (guiding scale for MDP-B) for the following concerns: (1) We observe that sctting the manipulation schodule and interpolation factor can both adjust the degree to which the layout is maintained in the edited image to some extent; (2) We want to find good default setting of interpolation factors for each manipulation; (3) When setting the interpolation factor 8 = 0 for MDP-3, this manipulation is equal to MDP-c, which we want to avoid in order to show the characteristic of each manipulation. We therefore empirically fix the interpolation factor of MDP-x,, MDP-c, P2P, MDP-3 and MDP-e, to be 0.7, 1, 1, -0.3, and 1, respectively. In general, lmar is ranging from to 5 while TM is set to be around 20 for each manipulation. We tune the manipulation schedule for each manipulation to obtain the desired editing result. We analyze and summarize how the methods work under various manipulation schedules in the Supplementary Materials.
Implementation. For text-guided editing, we test all the methods using the publicly available latent diffu- sion model Stable Diffusion D For class- class-guides guided editing, we use the conditional latent diffusion model Rombach ct al. (2022) trained on ImageNet Deng al. (2009). We test our manipulations on one NVIDIA A100 GPU. As no training and finetuning is required, each manipulation can be generally done within 10 seconds. As the inversion method we use is built on top of the DDIM sampler Mokady et al (2022) we also use DDIM sampler during the sampling process. However, our method can adopt other deterministic samplers.
Evaluation. We provide both quantitative and qualitative comparison for evaluation. For quantitative evaluation, We adopt three metrics evaluate the results. We collected around 90 real-world images and grouped them into different editing applications for evaluation. Part of the images examples are shown in Fig. 4 and Supplementary Materials. We adopt LPIPS Zhhag et al. 20188) CLIP score Hessel et al. (0022) and CLIP directional similarity Gal et al. (2021) for evaluation. Explanations of these metrics can be found in the Supplementary Materials. Despite from these metrics, which do not necessarily align with the human perspective, we conducted a user study to compare MDP-E; against baseline Prompt-to- torpptpp using figures shown in the Supplementary Materials. We conducted the study on MTurk and asked the participants evaluate based on three different aspects:
Image layout preservation: this metric is to evaluate if the edited image preserve the overall layout of the input image.
Image quality: this is to see if the visual quality of the edited image is good or not.
Image-text alignment: this metric is to evaluate if the edited image applied the prompt appropriately.

---

Published in Transactions on Machine Learning Research (09/2024)
Imput
MDP-r;
MDP-c
P2P
MDP-8
MDP-Et
"Cat" " "Tiger"
"Train" " to "Coach"
Add "Car" m
Add "Glasses" "
Remove "Graffiti"
Remove "Cakes" "
"Coco- late" to "Straw -berry"
"Woman" to "Kid"
"Corgi"
mixes "CofTee
Machine"
Figure 4: Text-guided local editing results.

---

Published in Transactions on Machine Learning Research (09/2024)
Input
MDP-1:
MDP-c
P2P
MDP-8
MDP-Et
"Grass" to "Library"
"Sky" "Runway"
"Spring" to "Winter" "
"Black "White" to "Colored"
"Photo" to "Oil Painting"
"Photo" to "Pencil Sketch" "
Stylize "City"
Stylize "Chair"
Figure Text-guided global editing results.

---

Published in Transactions on Machine Learning Research (09/2024)
4.2 Quantitative results
We show the quantitative results in Tab. 2 Results show that MDP-6 has the best consistency regarding the changes between the input image and edited images and the changes between the input condition and new condition. Although DiflEdit achieved the best LPIPS score, we argue that DiffEdit is an local editing method that only edits a localized region of an image. Therefore, the degree of changes in the edited images is usually lower. However, DiffEdit can fall short of global editing applications. We further argue that higher number of LPIPS or CLIP score alone does not necessarily mean a belter quality, as identical images can obtain almost infinite LPIPS, while images that are edited too much can also obtain very high CLIP score. heless, CLIP directional similarity, which takes imput image, edited image and text prompt into considcration at the same time, plays a better role for accessing image-text alignment as well as preserving original content. In this case, MDP-e; demonstrated the best quantitative evaluation results.
As those quantitative metries cannot fully aligu with the human judgement, we also show the user study results. In total, we could acquire 60 workers from MTurk to participate in our study. In detail, the edited images generated by MDP-6; were preferred over P2P in 60.6 % of the cases w.r.t layout preservation of the input image, in 67.9 % of the cases w.r.t. general quality of the output and in 73.0 % of the cases w.r.t image-lext alignment. We show more details of the user study in the Supplementary Materials.
4.3 Qualitative results
St
Input
C
Manipulation starts from carly to late
Figure 6: We change the background of the input image from "grass" to "library". We show the results of MDP-I,, MDP-c, and MDP-e, from top to bottom. The manipulation range TM is set to be 10, 20, and 20 respectively, which are the best settings for each manipulation. Manipulation e can do faithful edits across different manipulation ranges while the other two methods fail.
4.3.1 Local editing
We show examples of local edits for changing object, adding object, removing object, changing attribute, and mixing objects in Fig. [ The edits by MDP-3 are somewhat reasonable, but the overall layout is not well preserved. In general, all the other manipulations can do the edits guided by the text prompt while preserving the background of the input image. As the initial diffusion generation steps contribute to the layout of the generated image, we thus recommend to start the manipulation at the carly diffusion stage, usually la can range from 50 to 45, and TM can be ranging from 15 to 25. For MDP-I; and MDP-e,, editing can start at later steps, ie. Emax can be chosen to be smaller, as more layout is preserved during editing, Additionally, we observe some interesting findings that Prompt-to-Prompt fails for edits that remove objects. For the application of mixing objects, there is no universal standard of what the mixed object should look like, we provide more examples generated by each method in the Supplementary Materials.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Input
Dog
Foz
Wolf
Bear
an OY
rg regabo
Input
Plate
Clock
Lemon
Pizza
Input
Agaric
Purse
Teapot
Printer
Figure Class-guided image editing. We synthesize the input image using one class label, then use another category from ImageNet to edit the input image. We use MDP-6, to demonstrate the ability to do the class-got
class-guided guided editing.
4.3.2 Global editing
We visualize the results in Fig. 5 While in local editing MDP-z, and MDP-c can yield good results, in global editing they are either not able to do the edits, the overall layout is changed too much. In contrast, We observe that MDP-e, can produce very good results most of the time even when the strong baseline method Prompt-to-Prompt (P2P) fails. Further, we compare the editing results for MDP-x, MDP-c and MDP-e, under different ranges inject layout from the input image in Fig. G We observe that MDP-6, can create meaningful edits when the manipulation starts at carly or later stages. We therefore strongly recommend using MDP-e, when doing global editing, as it is more stable and can provide a variety of meaningful results. Here conjecture the reason why P2P fails to perform grained details. As we analyze, injecting original attention maps has weaker effect of preserving original layout. Therefore, P2P needs to inject attention maps for more sampling timesteps to preserve layout. However, as we show in Fig. 2 semanntcs are formed in the later sampling timesteps, more original semantics are also injected. That means the failure cases P2P, it either cannot preserve the original layout, or too much original semantics are leaking to the edited images. As for DiffEdit, as is a mask-bascd method, it does not perform well in the global editing applications compared to the local editing applications. We also provide more results in the Supplementary Materials.
edits the edited images are lack of fine-
4.3.3 Class condition models
We provide class-guided editing results using a class-condition diffusson model instead of a common texi- conditional model. We show results from MDP-e, in Fig. 7
5
Failure cases
We identify several typical failure cases of MDP-c, in Fig. g with comparisons Prompt-to-Prompt. For the first case, when we deform the object the input image such as changing the shape, neither of MDP-e, Prompt-to-Prompt faithfully perform the edits while preserving the layout of the input image. This
11

---

Published in Transactions on Machine Learning Research (09/2024)
kind of failure cases of deformation is also reported in the Prompt-to-Prompt paper. That means these two algorithms are better suited for performing edits without large changes to the object; in other words, the constraint to preserve the layout from the input image is relatively big in these two algorithms, which limits their ability to perform edits with larger changes. In another case MDP-e, cannot convert the red car into a yellow one, instead, Prompt-lo-Prompl can do the edit. We observe that some cases MDP-e, struggles transform the color of the object, especially the color that very different from the original one. We conjecture that by keeping the predicted noises from the input image path, lot of information about the color is also preserved.
Another common kind of failure cases is map-based image translation, such as converting segmentation map to a photo or & sketch to photo. All the manipulations we analyze in the design space preserve the color and texture of the input image to an extent, which is because of the nature of this kind manipulation. However, for map-based input images, the colors are usually only used to distinguish different parts of the objects, other than the real colors of the objects have. Therefore, the map-based image editing usually involves an auxiliary network additional training to extract the useful features in order to guide the editing (Voynov al. 2022 Avrahami et al. 2022a)
We also report another common failure case in Fig. which is result of the imperfections in image inversion. As all the manipulations in MDP for real images rely on image inversion, when the inversion algorithm cannot. invert the input image in a plausible way, the resulting edited image will most likely fail.
Iuput
Edit
P2P
MDP-e:
Input
Edit
P2P
MDP-e:
"Sitting"
"Red" to
"Jumping"
"Yellow
Input
Edit
P2P
MDP-c:
Input
Edit
P2P
MDP-cs
"Segmentation
"Sketch"
map"
to "Photo"
"Photo"
(c)
(4)
Figure 8: Typical failure cases of MDP-E, In (a) MDP-E, fails to preserve the layout of the input image; (b) MDP-6, fails to faithfully synthesize the yellow color. Tn (c) and (d), MDP-6, fails to do the map-based image editing, where the color and texture of the input map has semantic meaning.
Input
Inverted
Edit
MDP-Et
"Cat"
"Dog"
Figure MDP-6, fails when the inverted image does not reconstruct the input image in a plausible way.
6 Conclusion and Future work
We introduce generalized editing framework, MDP, which contains different manipulations that are suitable, their parameters, and the manipulation schedule. We highlight a new manipulation by oditing the predicted noise. The results show that this manipulation achieve high quality edits in challenging cases
12

---

Published in Transactions on Machine Learning Research (09/2024)
where previous work may fail. Our work also has multiple limitations. While analyzed many design choices, we cannot claim that our framework is exhaustive. Some new ideas will easily be integrated into our framework while others may be fundamentally different. We did not analyze editing operations that become possible using fine-tuning or modifications of the network architecture. Also, we rely on the reconstruction ability of inversion for real images. In some cases the inversion may fail to faithfully reconstruct the input. We also consider move the discussion of our framework into a continuous space.
7 Broader impact
Here, we discuss several potential impacts of our work broader context. Our paper uses publicly available Stable Diffusion as our diffusion model, which inherits the biases from its training dataset. The biases include but are not limited to race, ethical groups, gender, socioeconomic status, and culture stereotypes. A more diverse and inclusive dataset can reduce the biases. Moreover, bias detection and correction tools are important to apply to the output of the diffusion models or be integrated into the generation process.
Diffusion models can generate harmful content, such as deepfakes, misleading images, and inappropriate material. Implementing an effective filter is crucial for detecting and screening the outputs of these models. For cerlain lypes of content, like deepfakes, which may not easily caught by standard filters, more advanced methods should be employed to prevent misuse of the diffusion models.
Since framework edit images based text prompts, there is risk that it could be used to modify and distribute proprietary designs artworks without permission, potentially violating copyright laws. To prevent intellectual property rights infringement, it is necessary to develop tools for verifying the originality of input images.
13

---

Published in Transactions on Machine Learning Research (09/2024)
References
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. arv preprint arXiv:2211. 14305, 14905, 2022a.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. 2022 IEEE/C /VF Conference on Computer Vision and Pattern Recognition (CVPR). TEEE, jun 2022b. doi:
2wt.:5226688222.01100 URL /ttp:///////ooo.orr/////./ org/10. http::///bbb.a.r1000001011111112 2022. 01767
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion, 2023.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. ar preprint vrXiw:2302.07121, 2023.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffuusio models. arXie preprint arXiv:2112.03120 2021.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba, Semantic photo manipulation with a generative image prior. orXiv preprint ar Xio:2005.07727, 2020.
Manuel Brack, Patrick Schramowski, Felix Friedrich, Dominík Hintersdorf, and Kristian Kersting. The stable
artist: Stecring semantics in diffusion latent space. arXiv preprint arXiv:2212.0601 2022.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. Xrxiv preprint Xiv:2221 1.09800, 2022.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peekaboo: Text to image diffusion models are zero-shot segmentors. arXiv preprint orXiv:2211.1322, 2022.
Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excitee Attention-based semantic guidance for text-lo-image diffusion models. arXie preprinl N.N::2201.13826, 2023.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXio:2108.. 029338, 2021.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Woo Ha, Sunghun Kim, and Jacgul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7789-8797,, 2018.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-baase semantic image oditing with mask guidance. arXiv preprint arXiv:2210.11429 2022.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision ECCV 2022: 17th European Conference, Tel A Israel, October 23-27, 2022, Proceedings, Part XXXVII, pp. 88-105. Springer, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 2009.
Ieee,
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthe- sis. In Marc'A Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 8780-8794. 2021. URL
(Badaallessaatdaer7aasssatttee Abttffat-bbtttt.....mm html
14

---

Published in Transactions on Machine Learning Research (09/2024)
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Aduances in Newral Information Processing Systems, 34:19822 19835, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hicrarchical transformers. arXiv preprint arXiv:2204 14217, 2022.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scenee Scene-based text-to-image generation with human priors. In Computer Vision-ECCT 2022: 17th European
Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 89-106. Springer, 2022.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or Stylegan-nads: Clip-guided domain adaptation of image generators, 2021.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross atlention control. arXie prepriul XXiv:2208.01626 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. evaluation metric for image captioning, 2022.
Clipscorc: A referenco-free
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint orXim:200.11590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851. Curran Associales, Inc., 2020. URL Utl.httpp://prcceeiing..ee neurips.
pdf
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:230111109; 2023.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image trans- lation. In Proceedings of the European conference on computer vision (ECCV), pp. 172-189, 2018.
Phillip Tsola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-lo-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basod generative models. In Proc. NewrIPS, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irami. Imagic: bast- real image editing with diffusion models. arXiv preprint arXiv:2210.00 09.077... 2022.
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross- domain relations with generative adversarial networks. In International conference on machine learning. pp. 1857-1865. PMLR, 2017.
Gihyun Kwon and Jong Chul Ye. representation. arXiv preprint arXiw:2200.152: 2022.
based image translation using disentangled style and content
Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, and Jiaya Jia. Sdm: Spatial diffusion model for large hole image inpainting. arXic preprint orXiv:2212.0296 2022.
Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210. 16056, 2022.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 289 209,299,,40 2023.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pp. 11461 11471, 2022.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXic preprint arXiv:2108.010 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.0979, 2022.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistie image generation and editing with text- guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804. PMLR, 17-23 Jul 2022. URL https://proceedii wwtpang..mm.....m.. press/resss/vvee htto/iichhol10 c2w.choolaa........ html
Dong Huk Park, Grace Luo, Clayton Toste, Samaneh Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shapo-guided diffusion with inside-outside attention. arXiv preprint arXi::221 002100 2022.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. arXiv preprint orXiv:2302.008 08022, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsca Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with CLIP latents. CoRR, abs abs/2204.06115, 2022. doi: 10.48530/arXiv. 220....0......55.... URL
https://doi.org www.oorg//.00/. 48554/855iaiii
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CU //E//VF Conference on Computer Vision and Pullern Recognition (CVPR), pp. 106840 10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Floct, and Mohammad Norouzi. Photorcalistic text-to-image diffusion models with deep language understanding. CoRR, abs 11447 2022. doi: 10.48550/arXin 2148. 114. URL https:
//doi. org/10. 48550/855555. aaw.iv.iiii.i..i// 2205 11487
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representaatioss ICLR 2021, Virtual Event, Austria, May 9-7, 2021. OpenRe- view.net, 2021. URL http:p///pppppeeeeeeeeeeeeerests
Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffiusion features for text-driven image-to-image translation. arXio preprint orXiv:2211.1251 2022.
Andrey Voynov, Kfrr Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. ar preprint arXiv:2211.1974 2022.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. arXiv preprint or Xio:22211. 12446, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Ting-Chhn Ting- Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798 8807, 2018.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint or urXiv2212.05003, 2022.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-bane basod image editing with diffusion models. arXiv preprint arXiv:2211. 13227, 2022.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models, 2023.
URL htttp:://axxiii wwtps//rrgg/bbbb/2...o/o 05543
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effective- ness of deep features as a perceptual metric, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pp. 2223 2232, 2017.
17

---

Published in Transactions on Machine Learning Research (09/2024)
A More results
We show more comparisons between Prompt-to-Promp: and MDP-c, for cach kind of edits we describe in the main paper. We show the edits for local editing in Fig. 21 22
23 24 and 25 and for global editing
31 and 32 We also show more intermediate results for mixing objects in Fig. 26 We show that in many cases, ED can perform as good as or better than Prompt-to- coopttto-Prompt. Additionally, we also show editing results of MDP-c; using Stable Diffusion 2.1 as the base diffusion model in Figure 27 Noted that all the input images we use here are synthesized from Stable Diffusion 2.1 instead of real images. We also provided comparisons between DiffEdit and MDP-e; in Figure 28] As an editing method that only edits local regions in the images, DiffEdit has difficulty to perform global editing applications.
in Fig. 29
30
B Taxonomy of Image Editing Applications
In this subsection we describe small laxonomy for text-guided image editing applications that use to analyze our framework and compare it to previous work. To be noted that this taxonomy only includes some major image editing applications but is not an exhaustive one. Given either a real or synthetic image as input, we edit the image following a condition such as text prompt or a class label. We divide all common image editing operations into two categories: local editing and global editing. We further identify sub-categories for both local and global editing.
B.1 Local editing
Typically, we want to perform edits while keeping the overall image layout, e.g. background or the shape of the chosen object in the input image.
Changing object: change an object (or objects) in the image to another one. For example, if there a basket of apples, we may change it to a basket of oranges. Also, we may change the image of a dog to an image of cat.
Adding object: add an object that does not exist in the original image. For example, given an image of a forest, we add a car in that forest. Given a cat face, we can add a pair of sunglasses on it.
Removing object: remove object from the image. For example, can remove the eggs in a basket so that the basket becomes empty.
Changing attribute: change the attribute such color and texture of an object. Given a red bird, we may change it to a blue bird. We may change human portrait from young person to an old person.
Mixing objects: combine an object in the input image with another object. The two objects we may want to mix can have different semantics, egg. we can mix corgi and coffee machine, or a chocolate bar and a purse.
B.2 Global editing
For global editing, the overall texture and style of the image can be changed while the layout and semantics should be the same.
Changing background: change the background while keeping foreground object untouched. For example, we change rabbit on grass to a rabbit on the moon. We can also convert a black-and-white photo to be colored.
Stylization: stylize an object or a scone using an input style image. For example, we give an futuristic image as style input and we want the a generated image to have the same style.
hhtpps//ssaahilyyal/eeesss/t
1-release7-dee-2000
18

---

Published in Transactions on Machine Learning Research (09/2024)
In-domain transfer: edit the imput image by performing in-domain changes. For example, transfer photo of valley in summer to photo of a valley in winter, or we transfer a city during the day to a city at night.
Out-of-domain transfer: edit the input image with out-of-domain changes. For example, we transfer a photo of a valley an oil painting of valley or we change a portvoit of a human to cartoon character. We can also change black-and-white photo to colored photo.
C
Quantitative metrics
We adopt LPIPS Zhang al. (0018), CLIP score Hessel et al. (2022) and CLIP directional similarity Gal
al. 2021 quantitative metrics:
LPIPS: It can evaluate the distance between two images. We use LPIPS to evaluate if the edited image is similar to the input image, as to see if the edited image preserve the overall layout of the imput image.
CLIP score: can be used to evaluate the similarity between the image-text pair. this work we use CLIP score to evaluate the edited image is aligned with the new condition cBB) or not.
CLIP directional similarity: This metric is to evaluate if the changes in images align with the changes in text. In this work we adopt this metric to see if the changes in the edited image compared to the input image align with the changes in the new condition c compared to the input condition c(4),
D Preliminaries
Denoising diffusion probabilistic models. To train a conditional generative diffusion models, we con- sider the following objective,
min D. ||e fie--egle Ex cx,xx,,,)))?
(10)
where D is image dataset (could be raw image pixels latents obtained from image autoencoder), x is noised version of the image Xo, e is conditional embedding (e.g., text, class label, image) and c is a neural network paramcterized by 0. After training, we can sample new image given condition c with the commonly used DDIM sampler,
X -DDIM(X;, tt,
(11)
=Vat-1 fa(x:: c, t) + VI az-1 6a Ex(x,,
6; = cx(, c, and aa is a noise schedule factor as in DDIM. The sampling is to iteratively apply the above equation by giving initial noise Xy ~ (O. For brevity, we only consider DDIM as our sampler.
where fo(xe, x- Vaz
Image editing using pre-trained diffusion models. Our generalized editing framework MDP edits an image in its latent space also called noise space. Given an image Xo and the corresponding initial noise xy: modify Xy or the generating process and our aim is to obtain different x1 which suits needs. We summarize the symbols used in our later discussions here. One step of diffusion sampling can be represented
G co (x,, t).
as
We represent the intermediate results of the generating process at timestep
x;-1 = DDIM(X, t).
by X; = Gen(xx, c,t), given an initial noise xy and a condition c. We also represent all the intermediate outputs by ((x,, Et) e) (=pr......... GenPath cxr,
19

---

Published in Transactions on Machine Learning Research (09/2024)
Diffusion Inversion We can edit image using its latent representation. Diffusion inversion tries to revert the generating process, L.e,, given an image Xo, to find the initial noise x (and the intermediate noised images xz) which generates Xo. The first approach works by adding noise to Xo. The noise schedule is exactly the same as the training noisc schedule of DDPM. However, this process is stochastic and we cannot guarantee the obtained initial noise Xr can reconstruct Xo faithfully. The second approach is DDIM inversion based on the DDIM sampler in equation 13 We can calculate the imitial noise by applying the formula iteratively, X var+i [x, () + vI -r+0++1 Eg Ct), But when working with classifier-free guidance, the reconstruction quality is not satisfying. The third inversion method is Null-text Inversion Mokady et al.) (2022). It is based on the DDIM inversion and optimized for high reconstruction quality. In most cases, we will use the Null-text Inversion as our diffusion inversion method.
E
Algorithms
We show implementation details about MDP-E; in Alg. D MDP-c in Alg. 2 MDP-x, in Alg. 3 and MDP-8 in Alg. For real image editing, we use Null-text inversion to obtain Xy. For synthotic image editing, Xy is sampled from the standard Gaussian distributiin.
Algorithm MDP-6,
Require: [w.)t=-T.....)). Xr: c(A), c(A) (xx/4),((*)
GenPath (xr: c(1),))
(T.......))
(*) =x
XT
for in [T, T - 1....,1) } do
E, (B)
x; ,
(+)
= EB
6 (+) (1-we)) wi)E
(B) + w (1)
x DDIM (x("), (1)
end for
Algorithm MDP-c.
Require: (wi) (wrhe= XT, c(4), (B
(+) =XT
X1'
for / in [T, -T,T-1. - 1,, 1,.... 1) do
c, (a)
(1 w/ (B) w(c())
6, (+)
Es (x(") ,c("),
x DDIM (4) ((t)))
end for
Algorithm MDP-x,.
Require: (T...........)..x c(d), c(A), {xx111) (x()))((()) {(()))
GenPath (xr, c(4),))
X1 =XT
for in (T, T
1 ...,, 1)
do
(B+)
(t) (B)
x c
(B+) DDIM x C.
(+) Bxx)
x-1
(Bx)
(1)
x-_ (1 x
+ wixi1
end for
20

---

Published in Transactions on Machine Learning Research (09/2024)
staspe oeiina mage
Whth mugerve nes the right geile? Letto better light boteer
Figure 10: Screenshot of the interface for the user study.
Algorithm 4 3.
Require: (wr)t=pr....0...7) Xr, c(4), c(B)
X1T Xx
for t in {T,T - 1,....1} do
(1+) = 6D
Xi c(A)
6 (B+) En
x c(BB)
(t) = (1 - wi)E (Bx) + whe (1+)
x DDIM X ct
(t)
end for
F Availability of the initial condition
In practice, when editing real images, users need to provide c(4) as the initial caption for the input image. This caption is used during the inversion of the real image and guides the synthesis of the edited image. A pre- trained image captioning tool, like BLIP2, can be used to generate a caption for the input image. However, the generated caption may not always be optimal for producing the best editing results. For instance, we want to edit an image of forest Spring to look like Winter, including the keyword "Spring" in c(4) can offer more precise guidance for the diffusion models edit from "Spring" to "Winter". While it is possible to omit the keyword or even use empty text prompt, we found that specifying the keyword explicitly often results in more stable inversion and editing outcomes. However, an image captioner might not automatically generate a caption containing this keyword. Since both the input condition c(4) and the new condition cBB) are crucial for achieving satisfactory edits, future work could focus on developing efficient methods for generating captions tailored to text-guided image editing tasks.
G
User study
We show the interface of the user study presented to the participants in Figure 10 and the editing prompt in the middle column, along with two edited images in the left and right column, respectively. One of the edited image is from MDP-6, while another is from P2P. We randomly place either MDP-6 on the left or right side. For each example, we ask the user three questions which correspond to the evaluation aspects we discuss in the main paper:
We show the input image
Image layout preservation: "Which image better preserves the layout of the input image?? ?
Image quality: "Which image has the higher qualily?"
Image-text alignment: "Which image applies the prompt more appropriately?"
21

---

Published in Transactions on Machine Learning Research (09/2024)
H More Analysis in the Design Space
We observe that the manipulation schedule has a relatively larger influence on MDP-x, MDP-c, and MDP- 6g- For MDP-3 the linear facfor (guidance scale) has a much more important effect towards the controllability of the edited image. For Prompt-to-Promp (a manipulation for attention maps) the range of time steps to inject the self-attention maps strongly determines what the edited image will look like. Therefore, for MDP-x,, MDP-c, and MDP-E,, we explore four different manipulation schedules: constant, linear, cosine, and exponential (Fig. 11I. Given an integer timestep E [0, 50), we give the equations for the linear factor for constant schedule (wfoust), linear schedule (wfinear), cosine schedule (w and exponential schedule linear schedule (w
if Emin S mmax: else.
Wi
const
0
Amin
if tmin St 50,
linear
50
tmin
W
0
else.
(12)
cosiue
cos
50 Emin
if tmin 5 S 50,
0
clse.
w
exp
(-5
50 Emin
50
if Emn 5 50,
0
clse.
The results are shown in Fig 12
3
14
For MDP-3, we vary the guidance scale from -0.7 For Prompt-te to ro-Prompt we use the constant schedule to inject the
15
16
17
to 0.7 and show the results Fig 18 attention maps and show the results Fig 19 and Fig We only visualize one example for global editing as we show in the paper: given an airplane image imput, we change its background from "sky" to "airport runway". We only show this as an example because this example is challenging so can identify the ability of different methods. For Prompt-to- troottp, show additional example from local editing for better understanding of the method. We discuss the results in the following subsections.
For the parameters in the design space for each manipulation, we show a general recommendation in Tab and Tab4 which is summarized based on the examples we have tried. Note that this only considered a recommendation for general cases. For specific applications (and input images), improved results can obtained by fine Ane-tuning the parameters.
1.0
Linear lactor
Lincar factor
Cosine
Linear
Exp.
Asic
Timestep
Timestep
(b)
Figure 11: Linear schedule shown in (a), while linear, cosine, and exponential schedule are shown in purple,
orange, and green, respectively, in (b). For linear schedule, we fix the linear factor 1.0 while varying tmax and tuin by varying TM- For the other three schedules, we fix the tmax = 50 then vary scale factors and tmin-
22

---

Published in Transactions on Machine Learning Research (09/2024)
Suitable
Manipulation
fmax
TM
Linear factor
Other
or not
schedule
Rt Partly
[42, 48]
[15, 20]
Constant or others
< 1.0
C
Yes
[47, 50]
[15, 25]
Constant or others Can be 1.0 or smaller
The timesteps to inject self-attentic maps are critical [-0.8, -0.5] The guidance scale is very sensitive
P2P
Yes
Usually 50
40
Can be 1.0 smallerCan be 1.0 or smaller
B Partly
50
50
Does not matter
Ft
Yes
[44, 50]
[15, 25]
Constant or others Can be 1.0 or smaller
Table The recommended parameters in the design space for local edits.
Suitable or not
Manipulation
Imax
TM
Linear factor
Other
schedule
Zt Partly Partly
[40, 50]
[10, 15]Constnn or others
1.0
50
220, Constnat othersCan be 1.0 smaller
The timesteps to inject self-attention maps are critical Does not matter [-0.8, 0.5] The guidance scale is very sensitive
P2P
Yes
Usually 50
40
Constant or othersCan be 1.0 smaller
8
Partly
50
50
C
Yes
[44, 48]
[15, C5]stntt or othersCan be 1.0 or smaller
Table 4: The recommended parameters in the design space for global edits.
MDP-x:, MDP-c and MDP-6; While in most of the local editing applications and some of the global editing applications, MDP-x; can faithfully do the edits, for the example we show in the figures, all the examples generated by MDP-x, under different schedules fail. The edited image preserves the overall layout of the input image, however, the new semantics fail be injected. We conjecture that keeping the intermediate latent from the path when generating the input image will impose very strong condition to preserve the information from the imput image. For the result in the main paper, we fix the linear factors for MDP-x; to be 0.7 under all the edits. However, for a clear comparison with other two similar methods MDP-c and MDP-c, we fix the linear factors as 1.0, which results in that the new condition has no influence when bmin S 1 S mmax- That is also the reason why we empirically set the linear factor of MDP-x, to be 0.7 for all the results in the main paper, rather than 1.0 as the other two manipulations.
On the other hand, MDP-c and MDP-E; can inject the new semantics under the same schedules when linear factors all set to be 1.0 when fmax- For MDP-c, typically when fmax is smaller than 50 TM is smaller than 15, the new semantics can be injected into the edited image. However, the layout information from the input image is also lost. For MDP-ct, in terms of constant schedule, when Tar is 25 and fmax is smaller than 50, the edited image can both preserve the layout and incorporate the new semantics.
Empirically, we observe that MDP-x, is the strongest one to inject layout information, then is MDP-éa, while MDP-c the weakest one when using the same manipulation schedule. Theoretically, this observation is aligned with the diffusion generation formula:
X,-1 =DDIM(X;; E1,
(13)
-Volii folx,, + v1 11-00-1 6x(x,, e,t,
where fo(x,, c,t) X--VI-arc G, = cx c,t,, and a is a noise schedule factor as in DDIM. The Vos
sampling is to iteratively apply the above equation by giving an initial noise Xy (O. I). When setting the linear factors as 1.0, MDP-x, directly replaces the intermediate latent from the path when generating the input image, while MDP-6, replaces the predicted noise and keeps the previous intermediate latent when applying equation 13 and MDP-c only replaces the conditional embedding when predicting the noise.
We also observe that when fmax = 50, i.c., the manipulation happens right at the start of the whole denoising process, and the layout information can be injected more than when max is smaller. In addition, considering
23

---

Published in Transactions on Machine Learning Research (09/2024)
linear, cosine, and exponential schedule, for cosine schedule it can preserve the most layout formation, while the exponential schedule can preserve the least. This is also aligned with the design of the schedule, where the linear factor of the cosine schedule in every timestep is larger than which of the linear schedule and exponential schedule, which means in the cosine schedule larger amount of the layout information is taken into the consideration compared to linear and exponential schedule. These findings can also support that the initial timesteps during the generation process, especially the first denoising timestep TM = 50, have a great influence on the layout of the edited image. Injecting the layout information during the early generation stages is effective.
Considering the results we have obtained, we have several conclusions:
For the same manipulation schedule, MDP-x, injects layout information the strongest from the input image, then comes MDP-6,, while MDP-c the weakest. one.
For MDP-x, we recommend to set the linear factor smaller than 1.0 when using the constant schodule, or consider using schedule with decreasing linear factors during the generation process, such as linear, cosine, or exponential schedules. For MDP-c, we recommend using it only in local editing applications. For our proposed and highlighted MDP-Et, recommend to set tmax to be around 44 to 50 while TM to be around 20 to 25.
As the early stages during the diffusion generation process has larger influence the layout of the generated image, we recommend that for every kind of manipulation, we set the fax to be larger than 44.
MDP-3 Rocall the formula we use for MDP-8:
cl =a (xe, c(,) .t) +
(14)
B coo (xx, ((4), .t) -co co (xx, (5),
14 we want the output to have more characteristics from c(A) when generating using the new condition c (T) to preserve the layout. This is a bit different from the original classifier free guidance Ho & Salimans (022). where condition c(B) is an empty condition and 3 is a positive real number. In our specific image editing task, we find that when using a positive B (e.g., the first row in Fig. 18), the edited image will be too close condition c(4) while losing both the layout from the input image and the new semantics from condition c(B), Only when the guidance scale 3 is a negative number, can the new semantics be added into the edited image. In this situation, the guidance becomes a linear interpolation operation:
where 3 E R is called guidance scale. Intuitively, with equation
((() Ea (x), .) + ww) Ee (x,, (x))),e(x,,cc (B),)
(15)
where w = 1- 3 is a linear factor from 0 to 1, which means both condition () and condition (!)) contribute positively to the generation process. Overall, 3 has a larger influence over the edited image than varying the manipulation timesteps. We empirically find that B [-0.6, -0.8] can obtain better results.
Prompt-to-Promet We refer to the official implementation which also utilizes Null-text Inversion Mokady et al. (2022) to invert the real image and uses DDIM as the sampler. We observe that instead of replacing cross-attention maps, injecting self-attention maps has a bigger influence on the edited images. As in Fig[I] the effect of injecting self-attention maps is not obvious enough, we additionally show one local editing example in Fig 20 where when fixing the timesteps to inject self-attention maps, the edited image look almost the same under different timesteps to inject cross attention maps; while under different timesteps to inject the self-attention maps, the generated images look more different.
3
24

---

Published in Transactions on Machine Learning Research (09/2024)
We therefore suggest tuning of the timesteps to inject the self-attention maps. The more timesteps to do that manipulation, the more the edited image will look like the input image. In general, for global editing, where the changes from the imput image to the edited image should be larger, a smaller T'r can be considered compared to which for local editing. general setting for TM to replace the cross attention map like 20 or is enough.
25

---

Published in Transactions on Machine Learning Research (09/2024)
Emaass TM
5
10
15
20
23
50
48
46
44
42
Figure 12: Results of MDP-x, using constant schedule.
Type/TM
20
25
30
35
40
45
50
Linear
Cosine
Exponential
Figure 13: Results of MDP-x, using linear, cosine and exponential schedule.
26

---

Published in Transactions on Machine Learning Research (09/2024)
Emaass TM
5
10
15
20
25
50
48
46
44
42
Figure 14: Results of MDP-c using constant schedule.
Type/TM
20
25
30
35
40
45
50
Linear
Cosine
Exponential
Figure 15: Results of MDP-c using linear, cosine and exponential schedule.
27

---

Published in Transactions on Machine Learning Research (09/2024)
Emaass TM
5
10
15
20
25
50
48
46
44
42
Figure 16: Results of MDP-E; using constant schedule.
T
20
25
30
35
40
45
50
Linear
Cosine
Exponential
Figure 17: Results of MDP-6, using linear, cosine and exponent schedule.
28

---

Published in Transactions on Machine Learning Research (09/2024)
8/tmax Emin
50
40
50
30
50
20
50
10
50 0
0.7
0.3
0
-0.3
-0.7
Figure 18: Results of MDP-3 using constant schedule when varying guidance scale 3
29

---

Published in Transactions on Machine Learning Research (09/2024)
Self/ /Cross
50 30
50 20
50 10
50 0
40 20
40 10
40 0
50 40
50 30
50 20
50 10
50 0
Figure 19: Results of Soupp-to-Proopph using constant schedule when varying the timestpes to inject self- attention maps and cross-attention maps. We show the results when we vary tmax and fmin for either injecting self-attention maps or cross attention maps.
30

---

Published in Transactions on Machine Learning Research (09/2024)
/rof/Cross
50
30
50
10
4U
50-10
50-20
50-10
Figure 20: Additional results of Poomptt using constant schedule when varying the timestpes to inject sclf-attention maps and cross-attention maps. We show the results when vary tmax and tmin for either injecting self-attention maps cross attention maps.
Input
Edit
P2P
MDP-
Imput
Edit
P2P
MDP-
"Hore"
"Horse"
Zebra"
"Zebra" t
"Pizza" "Cake"
"Bread" "Cake"
"Cat" "Dog"
"Cat"
"Dog"
" "Bs"" to "Truck"
"Bus"
to TTrukk"
Figure 21: Results of changing object comparing Prompt-to-Prome and MDP-E:
31

---

Published in Transactions on Machine Learning Research (09/2024)
Input
Edit
P2P
MDP-ce
Input
Edit
P2P
MDP-6t
Add "Glasses"
Add "Glasses"
Add
Add
Giraffe"
Birdds"
APPLES
AL-ALES
Add Elephant"
Add Apples"
Add "Blloonss"
Add "Duck"
Figure 22: Results of adding object comparing to-pmpp and MDP-ct.
Input:
Edit
P2P
MDP-c.
Input
Edit
P2P
MDP-cr
"Sad" to "Happy"
"Sad" to "Happy"
"Old" "Young"
"Woman" "Kid"
"Chocolate" T to "Strawberry"
"Stawberry"
to "Candles"
M
"Cloth"
"Fluffy"
"Plastic"
"Smooth"
Figure 23: Results of changing attributes comparing Trompt-to-Prompe and MDP-é:
32

---

Published in Transactions on Machine Learning Research (09/2024)
Inpot
Edit
P2P
MDP-ct
Input
Rdit
P2P
MDP-6
Remove "Eggs"
Remove
"Fruits"
Remove "Fruits"
Remove "Popple"
Remove "Boats"
Remove "Cat"
Remove "Graffiti"
Remove "Stuffs" "
Figure 24: Results of removing object(s) comparing Prompt-to-Prompt and MDP-c.
Input
Edit
P2P
MDP-c.
Iuput
Edit
P2P
MDP-c:
"Tower" Mixes
"Watermelon" Mixes "Pillow"
"Dress"
"Butterfly" Mixes "Handbag"
"Chocolate"
Mixes "Purse"
"Flower"
Bear"
Mixes
Mixes "Kettle"
"Handbag"
"Pumpkin" Mixes
"Strawberry" " Mixes "Diamondd"
"Clock"
7
Figure 25: Results of mixing objects comparing Prompt-to-Promp and MDP-Er.
33

---

Published in Transactions on Machine Learning Research (09/2024)
MDP-ét
P2P
MDP-E:
8
P2P
Figure 26: More intermediate results of mixing objects comparing MDP-c, and P2P. In the first two rows mix the input "corgi" with "colfee machine". In the second two rows mix the input "pumpkin" with "clock".
Input
Edited
Input
Edited
"Chocolate"
Add "UFO"
to "Strawberry"
"Sea" to "Moon"
Remove "Conference room"
"Photo" to "Children's drawing"
"Spring" to "Winter"
"Corgi" mixes "Coffee machine"
Stylize
"Fox"
Figure 27: Editing results of MDP-E, using Stable Diffusion 2.1. Note that all the input images are synthesized from Stable Diffusion 2.1.
34

---

Published in Transactions on Machine Learning Research (09/2024)
Input
DiffEdit
MDP-c-
Input
DiffEdit
MDP-6
"Car" to "Tiger"
"Sunny" to "Rainstormy"
Remove "Fruits"
"Womman"" to "Kid"
"Photo"
"Grass" to "Library"
"Children's drawing"
"Photo" to "Pencil Sketch"
"Pumpkin" Mixes "Clock"
7
Figure 28: Comparison between the results of DiffEddit and MDP-Et
Input
Edit
P2P
MDP-6
Input
Edit
P2P
MDP-c
"Grass"
"Beach"
"Road"
"Mountain"
"Grass" "Roof"
"Ground" to "Boat" "
"Gruss" to "Lake"
"City" to Moonttain"
"Mountain" to "Mars" "
"Sea" " to "Monntain"
Figure 29: Results of changing background comparing Prompt-to toapp-tt-Prrompt and MDP-c.
35

---

Published in Transactions on Machine Learning Research (09/2024)
Input
Edit
P2P
MDP-G
Input
Edit
P2P
MDP-6
"Spring" to "Winter"
"Spring" to "Winter"
"Winter"
"Winter" "Spring"
"Spring"
"Colored"
Sunny
to Blackkaadk white"
Rainstormy"
"Evening" "Morning"
"Morning"
"Evening"
Figure 30: Results of in-domain transfer comparing Prompt-to-Prone and MDP-ct.
Input
Edit
P2P
MDP-c.
Iuput
Edit
P2P
MDP-c.
"Phota" to *Children's Painting"
"Photo" to "Children's Painting
VEA
"Photo" to "Watercolor Painting"
"Watercolor Painting" "Photo"
"Photn" to Immpecssooism Painting"
"Photo"
"Sketch"
"Photo" to "Abstract Painting"
"Oil Painting
"Photo"
Figure 31: Results of out-domain transfer comparing Prompt-to-Prompt and MDP-Er
36

---

Published in Transactions on Machine Learning Research (09/2024)
luput
Edit
P2P
MDP-c.
Input
Edit
P2P
MDP-c.
Stylize "Bus"
Stylixe "Flower"
Stylize "Ice-cream"
Stylize " "Book"
Stylize Hammerr
Stylize "Cells" "
Stylize "Coral"
Stylise "Ginl"
Figure 32: Results of stylization comparing Prompt-to-Promns and MDP-E.
37


Reviews:
[{"review_id": "u4Gv2wibrd", "reviewer": "Reviewer_QNkA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper attempts to unify some important diffusion-based image editing methods into a single framework.\n\nWeaknesses:\n\nMajor issue:\n1. The claimed contribution in this work appears largely exaggerated. Specifically, the contribution claimed mainly includes \"a framework for a generalized design space ... that includes multiple existing methods as special cases” for diffusion-based image editing, consisting of five different methods: (1) Interpolating denoised outputs, (2) Interpolating condition embeddings, (3) Change cross attention maps, (4) Classifier-free guidance, and (5) Interpolating predicted noise. Among these parts, (1), (3), and (4) are clearly existing works, as also suggested in this paper. Actually, (2) is also a previous finding, for example in [1,2]. Only (5) is a newly introduced algorithm. In other words, only one (or arguably, two) of the five methods are not from existing works, and the other methods are re-implementations of previous works. If more than half of the design space consists of existing methods, they cannot be claimed as merely special cases of a newly proposed framework.\n2. For the proposed method (5), it is important to discuss its differences with method (4) classifier-free guidance theoretically and carefully analyze the empirical differences in the paper because they are very similar. To start with, in method (4), the diffusion noise is a linear combination between  $\\epsilon^{(A*)}$ and $\\epsilon^{(B*)}$, while in method (5), the diffusion noise is an interpolation between $\\epsilon^{(A)}$ and $\\epsilon^{(*)}$. The only difference, if I understand correctly, is the noises used for interpolation. How does this difference lead to significant quantitative and qualitative differences in the experiment?\n3. Evaluation results: The experiment does not demonstrate the advantages of method (5). Specifically, from Table 2, the quantitative results show that different methods have their own advantages, and the qualitative results also do not show significant benefits. More importantly, in Sec. 4.2, the paper claims an advantage in user studies and states that the details of the user study will be in the Supplementary materials, but I couldn’t find them there. The description of the user study in the main text is ambiguous and lacks many details, and it is not clear whether \"our method\" refers to method (5) or the whole framework. \n\nMinor issue:\n1. Some of the discussions in the related work section are misleading. For example, (a) In the “Image editing with diffusion models” section, paragraph 2, the blended diffusion and blended latent diffusion are organized as requiring a finetuning process. From my understanding, these two works do not require finetuning of diffusion models and should therefore be placed in the third paragraph.  (b) For prompt-to-prompt, the paper mentions its drawback as “prompt-to-prompt always requires a prompt together with an input image” and claims that “our edits do not require prompting for an input image.” However, in the methods section, the paper states that the proposed method will need a text-condition $c^{(A)}$ for null-text inversion. There is a contradiction: Does your method require a text prompt? I think the answer is yes, and the claims in the related work section are incorrect.\n2. The experiments are done with stable diffusion 1.4, which is an outdated model. Why not use the latest stable diffusion model?\n3. The proposed method requires proper selection of $t_{max}$ and $t_{min}$, which could be tedious.\n4. Many potential typos. For example: (a) Figure 1 is not referred to in the main text; the text that mentions Figures 2 and 3 lacks “Fig.” (b) On page 6, when quoting equations, the text uses “Eq. Equation.” (c) In Table 2, the number 0.1851 seems wrongly highlighted; it is not the highest number. (d) On page 6, first paragraph, it says “we interpolate the predicted noise xt(A) …” while in the paper, $\\epsilon$ is used to refer to noise, and $x_t$ refers to the latent image. (e) Figure 6 is inconsistent with the text: In Figure 6, the $\\epsilon$ is copied from the $\\epsilon^{(A)}$, while in the text, $\\epsilon$ is interpolated between $\\epsilon^{(A)}$ and $\\epsilon^{(B)}$. \n\n[1] Imagic: Text-Based Real Image Editing with Diffusion Models\n[2] Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work discusses a unified diffusion-based image editing framework, consisting of a series of important existing diffusion-based image editing methods. It then proposes a new method for image editing, which interpolates noises conditioned on different prompts at inference time. This work compares the proposed method with existing editing methods both quantitatively and qualitatively, and discusses the impact of different hyperparameters, as well as the limitations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please address my concerns outlined in the previous weaknesses section, particularly the questions raised in the “Major Issues” part. Additionally, please carefully proofread the paper to enhance its readability.", "broader_impact_concerns": "I do not have any concerns regarding the broad impact section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9s91DyaF6P", "reviewer": "Reviewer_pdgE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nGood quantitative and qualitative experimental results.\n\n**Weaknesses:**\nIn general, the presentation of the paper is poor–-the citations appear strangely, multiple instances of cross-references are handled oddly, and there are formatting problems throughout. Further, there are grammatical errors and typos mixed throughout the paper.\n\nThe caption for Table 2 is uninformative, and there is not much discussion of these results. The CLIP directional similarity for DiffEdit is bolded, but it is actually the lowest of the methods. Figures 4 and 5 do not show any results from DiffEdit, while Table 2 includes these results. \n\nFormatting problems:\n- Citations always appear as: author (year)\n- The text in Table 1 is too small\n- References to figures/tables often just show as a number, while references to equations show as:  “Eq. equation [equation number]”\n- There is a dangling period after the sentence following equation 3\n- Equations 5 and 6 are split over two lines when there is space for them to fit on a single line\n- Figure 4: “Chocolate” to “Strawberry” row of images is not aligned with the rest\n- Figure 5: Stylize “Chair” row of images is not aligned with the rest\n- Figures 12, 14, 16, etc. are too wide", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a generalized editing framework, MDP that encompasses five different types of manipulations: intermediate latent, conditional embedding, cross attention maps, guidance, and predicted noise. The MDP framework in general enough to encompass existing methods for image editing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KtxVGr9DKP", "reviewer": "Reviewer_m17B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nS1) The paper is easy to follow. Most of the relevant information were quite clear even from the first read.\n\nS2) The more general topic of generative models is quite important and relevant right now. \n\nS3) The paper places the literature into a more general framework \n\nS4) The paper shows that this framework can \"produce\" other, useful solutions to the task and the proposed method seems to work quite well\n\nWeaknesses\n\nW1) The paper is not discussing the ethical issues of generative models\n\nW2) What happens if you don't have the condition c(A) available? Will this still work if you generate the c(A) from x0(A) through a captioner?\n\nW3) minor typos, convensions, or notation choices", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper sets a general framework for the training-free text-guided image editing task. It shows that a sufficient amount of literature methods are special cases of that framework and it propose a novel method from that framework that achieves high quiality results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Important for acceptance:\n\n1) I think a discussion about the ethics of generative models should be included in the manuscript.\n\n2) Also the availability of the initial condition should be discussed either with experiments or by arguing in the paper why this is not needed\n\n\nminor typos, convensions, or notation choices:\n\n1) Although the acronym MDP can be derived from the title, it is not defined in the text\n\n2) In page 4 in the paragraph \"Intermediate denoised output\", second line: is x0(B) supposed to be c(B)?\n\n3) In my experience \"lerp\" is quite unconventional as a math operation, would you consider to substitute lerp(A, B, w) with (1-w)A+wB?\n\n4) References to tables or figures are not stating if they are for a table or for a figure. (page 3 second to last line \"experiment in 2\", page 6 before section 4 \"method in 3\", page 7 first line of 4.2 \"results in 2\")", "broader_impact_concerns": "As I stated in W1 discussion about ethics is extremely important. The topics that need to be addressed are:\n\n1) Biases and discrimination from the datasets of the pretrained models\n\n2) Possible use for missinformation from the end user\n\n3) Possible exploit of intellectual properties from the industry", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework to unify several diffusion-based image editing methods. Based on this framework, this paper identifies a potentially overlooked image editing technique. The paper compares the new editing technique with existing methods and shows that it achieves better quality in many scenarios. The reviewers generally acknowledge the contribution of the unified framework, but they also raised concerns before the rebuttal period, such as over-claimed contributions (QNkA), ethical issues (m17B), and typos (all reviewers). During the rebuttal, the author made a good effort in addressing these concerns by revising contribution claims, adding discussions, and fixing the typos. Reviewers QNkA and m17B are satisfied with the modifications after the rebuttal. Additionally, reviewer pdgE provided important comments about the novelty and ethical implications. It is worth mentioning that novelty is not a necessary criterion for TMLR’s acceptance. Furthermore, the AE checked the discussion threads and found that the author provided ethics discussions in the thread for reviewer m17B. As such, the AE recommends the acceptance of this paper with recommended modifications: (1) Please properly add the ethics discussion and algorithm comparison discussion to the camera-ready paper; (2) Please fix the inconsistency between Figure 3 and Equation 9.
Decision: Accept (poster)