PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code
Authors: Xuan Ju
Ailing Zeng
Yuxuan Bian
Shaoteng Liu
Qiang Xu
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
image editing
diffusion inversion
Primary area: generative models

Abstract: Text-guided diffusion models have revolutionized image generation and editing, offering exceptional realism and diversity. Specifically, in the context of diffusion-based editing, where a source image is edited according to a target prompt, the process commences by acquiring a noisy latent vector corresponding to the source image via the diffusion model. This vector is subsequently fed into separate source and target diffusion branches for editing. The accuracy of this inversion process significantly impacts the final editing outcome, influencing both essential content preservation of the source image and edit fidelity according to the target prompt. 
Prior inversion techniques aimed at finding a unified solution in both the source and target diffusion branches. However, our theoretical and empirical analyses reveal that disentangling these branches leads to a distinct separation of responsibilities for preserving essential content and ensuring edit fidelity. Building on this insight, we introduce “PnP Inversion,” a novel technique achieving optimal performance of both branches with just three lines of code. To assess image editing performance, we present PIE-Bench, an editing benchmark with 700 images showcasing diverse scenes and editing types, accompanied by versatile annotations and comprehensive evaluation metrics. Compared to state-of-the-art optimization-based inversion techniques, our solution not only yields superior performance across 8 editing methods but also achieves nearly an order of speed-up.

Published as a conference paper at ICLR 2024
PNP INVERSION: BOOSTING DIFFUSION-BASED EDIT- ING WITH 3 LINES OF CODE
Xuan Ju Ailing Zeng"t, Yuxuan Bian', Shaoteng Liu', Qiang Xu
The Chinese University of Hong Kong (CUHK) International Digital Economy Academy (IDEA) fx2u22, st stliull, qxu Bese. cuck. edu. hk {zengai l edu. en
A. white horse nouning in the field Watercolor of white horse running in the field
Structure Distance
27.41
refowrs 20.30 a0%.
32.00
refowrs 38,75 151)
35.84
solours 21.53 ai%
45.68
vélonrs 25 er%
A cat sicting on a woodan chutr
dog sicting on wooden chatr
Bockground PSNR
25,77
wionrs 19,90 191-
21.96
soOnns 23.63 Mh!
21.05
MIQUTS 23.40 i1s)
31.81
HrQuIS 24.85 smi
Biue lavender in blue and white vase
Purple lavender in a 6lse and white vire
Bockground PSNR 1 21.76
refowrs 28,84 201-
21.32
w-lowrs 21,00 thi
23.71
MiQurs 27,21 1967
18,00
acGnus 18.19 1h:
(a) Source
P2P
(c) MasaCtel
(d) P2P-Zero
(c) PnP
Figure 1: Performance enhancement of incorporating PnP Inversion into four diffusion-based editing methods across various editing categories (from top to bottom): style transfer, object re- placement, and color change. The editing prompt is displayed at the top of each row, which in- cludes (a) the source image, the editing results of (b) Prompt-to-Pront Prompt (P2P) (Hertz et al. 2023
(c) MasaCtrl (Cao al. 2023), (d) pix2pix-zera (Parmar al. 2023). and (e) plug- and--lad (Tu- manyan et al. 2023). Each set of results is presented: the first column w/o PnP Inversion (Null-text inversion for P2P. DDIM Inversion for the others), and the second column w/ PnP Inversion. In- corporating PnP Inversion into diffusion-based editing methods results in improved image structure preservation (enhancement of the structure distance metric) for full image editing and enhanced background preservation (increased PSNR metric values in the background, i.e., areas that should remain unedited) for foreground editing. The improvements are mostly tangible, and we circle some of the subtle discrepancies w/o PnP Inversion in red. Best viewed with zoom in.
ABSTRACT
Text-guided diffusion models have revolutionized image generation and editing, offering exceptional realism and diversity. Specifically, in the context of diffusion- based editing, where a source image is edited according process commences by acquiring a noisy latent vector corresponding to the source image via the diffusion model. This vector is subsequently fed into separate source and target diffusion branches for editing. The accuracy of this inversion process significantly impacts the final editing outcome, inftuencing both essential content preservation of the source image and edit fidelity according the target prompt. Prior inversion techniques aimed at finding unified solution in both the source and target diffusion branches. However, theoretical and empirical analyses reveal that disentangling these branches leads to a distinct separation of respon- sibilities for preserving essential content and ensuring edit fidelity. Building on this insight, we introduce "PnP Inversion." novel technique achieving opti- mal performance of both branches with just three lines of code. To assess im- age editing performance, present PIE-Bench, editing benchmark with 700 images showcasing diverse scenes and editing types, accompanied by versatile annotations and comprehensive evaluation metrics. Compared to state-of-the-a optimization based inversion techniques, our solution not only yields superior per- formance across 8 editing methods but also achieves nearly an order of speed-up.
target prompt, the
"This work was done when Xuan Ju was intern at IDEA.
"orresponniig author.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
w/ & w/o guidance
guidancee
guidance
Source Lateat
(ooueiistatt
0= sounsae
idral inversian Latorr
Essential Content Preservation
Perturhed DRIM Inverdea Latear
12
(mis guidance) Source Prompi Forward Latent
louree
Nall-Test Text SiveeriinnSyyleeiffuiiioo
Source
Prompt Forward Latent
(ere paldiazce) Target
wi guidance
Prorept
(ni puidunce) Target Pronue Fornard Loteet
diseniangle
Forward
Brach Birevtien
Edit Fidelity
Inverse
Intersccioon
Order Operation
DDLM Inversion
Negative-Prompl Inversion
Pul Inversion
Figure 2: Comparisons among different inversion methods in diffusion- editing. We sume a 2-step diffusion process for illustration. Due to nonexistent of ideal 2/. common practice uses
with perturbation. Diffusion- DDIM Inversion (Song et al. 2020p to approximate z/. resulting in 21" A
based editing methods start from the perturbed noisy latent 2 and perform DDIM sampling in source and target diffusion branch, further resulting in the distance shown the figure. Null-Text Inversion and StyleDiffusion optimize a specific latent used in both source and target branches to reduce this distance. Negative-Prompt Inversion assigns the guidance scale to 1 to decrease the dis- tance. contrast, PnP Inversion disentangles source and target branches in editing. By leaving the target diffusion branch untouched, PnP Inversion retains the edit fidelity. By directly returning the source branch to z PnP Inversion achieves the best possible essential content preservation. We numbers to mark operation step order, where solid circles steps added by PnP Inversion.
Text-guided diffusion models (Rombach et al. 2022 Ramesh et al.) 2022) have become the main-
stream image generation technique, praised for their realism and diversity. As the noise latent space of diffusion models (Meng et al., 2022, Kawar et al. 2023 Hertz et al. 2023 Balaji et al. 2022] Liu et al., 2023a) possesses the capacity to retain and modify images, we can perform prompt-based editing with diffusion models, where a source image is edited according to target prompt. The common practice is to maintain two diffusion branches: one for the source image and the other for the target image. By carefully exchanging information between the two branches, the essential content in the source image while achieving edit fidelity according to the target prompt. However, such manipulations in each diffusion step) corresponding to the source image available. When editing images without known latent space, have invert the diffusion model obtain their latent vectors first.
preserve
only straightforward when the diffusion latent space (noisy latent
While DDIM inversion is effective for unconditional diffusion (Song et al. 2020 Dhariwal & Nichol 2021), much of the research (Hertz al.) 2023 Han al., 2023) has centered invert- ing the diffusion process with conditional inputs. This is driven by the significance of conditions in applications like text-based image editing. However, introducing conditions undermines DDIM inversion quality, as evidenced in Figure With the advent of Null-Text Inversion (Mokady et al., 2023). a prevailing consensus (Dong et al.] [2033 Li et al.) 2023b) has emerged: achieving su- perior inversior[] necessitates rigorous optimization. Methods that forgo such optimization, such as Degative-Promph Inversion (Miyake et al. 2023), compromise editing outcomes. this paper, we challenge this prevailing wisdom, posing two fundamental questions: What exactly are these optimization- based inversion methods truly aiming at? And, are such optimizations indispens- able for diffusion- based image editing?
As illustrated in Figure 2 prior optimization-ban based approaches strive to minimize the distance be- tween z5 and Fre 1 Feo by indirectly tweaking the generation model's 's input parameters. Given the magnitude of the optimization network, like UNet, and the impracticality of prolonged optimization durations, these methods often optimize the target latent with just a few iterations. This results
A more suitable name for them would be inversion correction techniques. But we follow previous works and call them inversion techniques.
2

---

Published as a conference paper at ICLR 2024
in learned latent with discernible gap between 20 12 learned latent does not appear during the generation models' training process, deviating from the pretrained diffusion model's original input distribution. Such forced input assignments hinder the model's generative capacity, compromising the integrity of both the source and editing branches.
Fig and the original 20 Moreover, the
In this work, we delve into the intricacies of text-based inversion, providing a thorough analysis of existing techniques. Our theoretical and empirical analyses demonstrate that the exhaustive opti- mization in prior techniques is, counterintuitively. not only far from optimal but unnecessary. Intro- ducing PnP Inversion (Plug-and -lay Inversion), our approach offers simple yet potent inversion solution for diffusion- baased editing. The essence of PnP Inversion lies in two primary strategies: (1) disentangle the source and target branches, and (2) empower each branch to excel in its designated role: preservation or editing. Specifically, the source branch in PnP Inversion rectifies the deviation path directly, using only 3 lines of code. This addresses the challenges seen in carlier approaches: (1) undesirable latent space distances affecting essential content preservation, (2) misalignment in the generation model's distribution, and (3) extended processing times. For the target branch, we keep it unchanged, ensuring the best possible edit fidelity in line with the target prompt.
Addressing the lack of standardized benchmarks for inversion and editing, we introduce PIE-Bench (Prompt-based Image Editing Benchmark) with 700 images from diverse scenes, spanning 10 edit- ing categories. Each entry consists of source prompt, target prompt, editing directive, edit subjects, and hand-annotated editing mask. To assess PnP Inversion and benchmark it against existing tech- niques, we employ 7 metries emphasizing both essential content preservation and edit fidelity. Com- pared with 5 inversion methods with Prompt-to- toorroppttt editing, PnP Inversion outperforms them, enhancing essential content preservation by up to 83.2% Structure Distance and edit fidelity by up to 8.8% in Edit Region Clip Similarity, while achieving nearly an order of editing speedup over optimization- based inversion methods. Morcover, across 8 editing approaches, PnP Inversion boosts content preservation by as much as 20.2% and edit fidelity by up to 2.5% relative to their baseline contigurations. Visualization results are shown in Figure
2 RELATED WORK
Diffusion- based image editing aims to manipulate images with diffusion models using given instruc- tions such as natural language descriptions (Hertz et al. (002)), dragging (Shi et al. 2023). and semantic masking (Meng et al.) 2022). This involves two primary concerns: (1) edit fidelity: ensure the editing aligns with the provided instructions; and (2) essential content preservation: inverse the images, particularly the regions that do not require modification, into diffusion latent space while ensuring accurate reconstruction during the editing procedure. Details are in supplementary files.
For edit fidelity, previous methods perform editing roughly through four ways: (1) end-to-end edit-
ing model (Brooks et al., 2023) Kim et al., 2022] Nichol et al. [022]] Geng et al. 2023) that trains
end-to-end diffusion models to edit images (2) latent integration (Meng et al. 2022 Avrahami
jet al., 2022] 2023 Couairon et al. 2023] Zhang et al. 2023b] Shi et al.] 2023 Joseph et al. 2023)
that inserts editing instruction through the noisy latent feature in target diffusion branch to source
diffusion branch. (3) attention integration (Hertz et al., 2023 Han et al.
2023 Parmar et al. 2023]
Cao et al. 2023 l'uaayyan al. 2023] /Zhang et al.; 2023a; Mou al. 2023) that fuses attention
map connecting the text and image in the source and editing di diffusion branch, and (4) target embed-
ding (Kawar et al., 2023 Cheng et al. 2023
Wu et al. 2023 Brack al., 20033 Issaaan & Passos]
2023 Valevski et al. 2022 Dong et al.
2023
Wu & De la Torre 2022 2023) that aggregates editing
information of the target branch into an embedding and then insert it to source diffusion branch.
For essential content preservation, previous methods tried retain the source image's feature through (1) overfit the editing image (Kawar et al. 2023 Shi et al., 2023) so that editing will not make massive changes to the image content, (2) DDPM/DDIM inversion variation (Miyake et al.]
2023 Huberman-Spiegelglaan et al. 2023 Wallace et al. 2023) to strengthen the source image's in-
fluence on both the source and target branch by modifying DDPM/DDIM inversion formulation, (3)
attention preservation (Mou et al., 2023 Cheng et al. 2023 Cao et al.) 2023 Parmar et al.) 2023
Tumanyan et al.) 2023 Hertz et al.| 2023 Qi et al. (022) that retains the attention map feature of
the source diffusion branch during attention map fusion of the source and target branches, and (4)
source embedding (Mokady et al.) 2023] Dong et al. 2023 Li et al. 2023b] Gal et al., 2022; Fei
eet al. 2023) Huang et al.](223) that absorbs the background or image information to an embedding
and use this embedding to reconstruct essential content of the source image.
"Code is available at wttpttts: / / wtt.:l.ccccccucceeeeeee 1acpourreeaaaaaaa PnP Inversion
3

---

Published as a conference paper at ICLR 2024
3 PRELIMINARIES
This section will first introduce DDIM sampling and classifier-free guidance commonly employed in Diffusion Models. Then, we will delve into the issues arising from the utilization of classifier- guidance and DDIM sampling in editing, and show how previous works address these challenges.
Diffusion Models. Diffusion models aim map a random noise vector ør series of noise samples 2ς and, finally, an output image or latent 2g by adding Gaussian noise c step-by-step, where t ~ [1, T] and T is the timestep number. To train diffusion models, first sample 2g from real image following equation] | where E ~ (0, and a is hyper-parameter
= 04 + vI Org
Then, denoiser network eg optimized predict <(2t, () with the objective:
min en MUl,a ||e (at, t)ll
To generate images from given 27, we employ the deterministic DDIM sampling (Song et al. 2020
(V-L
2--1 Va--1 Vai
2t + var-1
(Zz, ()
(3)
a--1
DDIM Inversion. Although diffusion models have superior characteristics in the feature space (Balaji et al.) 2022, Dong et al.) 2023; Feng et al. 2022) that can support various down- stream tasks, similar to GAN-based models (Xia et al. 2022). , is hard to apply them to images in the absence of natural diffusion feature space for non-generated images. Thus, technique invert- ing s0 back to žy (i.e., z+) is necessary. To address this, a straightforward inversion technique known DDIM inversion commonly used for unconditional diffusion models, predicated on the presumption that the ODE process can be reversed the limit of infinitesimally small steps.
ar
va
-V-----
1
(2(-1.t-1
(4)
0:-1
a1-1
However, in most diffusion models, this presumption cannot be guaranteed, resulting in a perturba- tion from of to zę" in equation (3) equuation4 Figure
2 Consequently, an additional perturbation
L to =f /12 arises when utilizing equation to sample an image from 2/," shown in Figure
from 21"
Classifier-free Guidance. To insert conditions, Ho et al. (Ho & Salimans 2022) present classifier- free guidance, which predicts noise both unconditionally and conditionally, then mix them together:
(5) where is the guidance scale, C is the condition (embedding of text prompt in task), and Ø is the null condition (embedding of null in our task). This further leads to another perturbation from F. 2, 1 2, 1 due to the destruction of the DDIM process as demonstrated in Figure
co (22, f, C, = w cg (zt, t, C) (1-w) w) (2e, t)),
Previous Inversion Techniques. Currently, the predominant inversion technique employed for reducing the adverse impact caused by DDIM inversion and classifier-free guidance is optimization- based methods, such Null-Text Inversion (Mokady et al. 2023) and StyleDiffusion (Li et al. 20236). Alternative inversion techniques, such as Edit Friendly DDPM (Huberman-S Suabermann-Spieelghaae al. 2023), Regative-Prompt Inversion (Miyake et al.) 2023), and EDICT (Wallace et al.) (003)), exhibit instability in editing outcomes in both essential content preservation and edit fidelity. The qualitative and quantitative results in our experiment further corroborate this instability.
Optimization-basel inversion methods use specific latent variable to reduce the difference between F.. and 2/-_ For example, Null-Text Inversion revises equation 5 to equation 6 and learns the 2,
specific latent variable by gradient propagation using the loss function 2, variable will be further used in both the source and target branches in editing.
Foy 2 This learned
eg (zx, 4, w eg (Ze, 1, C.C) + w) egg (Et, variable)
(6)
4

---

Published as a conference paper at ICLR 2024
4 METHOD
4.1 MOTIVATION
We explain our motivation by raising and then answering two questions.
Why do optimization-based methods perform better among previous inversion methods?
Edit Friendly DDPM (Huberman- Saatinnan-pieggllaass et al. 2023) proposes an alternative latent noise space by changing the DDPM sampling distribution to help reconstruction of the desired image. Negative- Prompt Inversion MMyake et al.) 2023) assigns conditioned text embedding to Null-Text embedding and thus maintains guidance scale of 1 to reduce the deviation in editing. EDICT (Wallace et al., 2023) maintains two coupled noise vectors to invert each other in an alternating fashion for image reconstruction, which reduces the editability of diffusion models. Compared with these inversion techniques, optimization-basee inversion (Mokady et al.) 2023 Li et al. 2023b Dong et al. 2023) does not influence the distribution in DDIM sampling (compared to Edit Friendly DDPM), retains enough guidance for text conditions (compared to Negative-Prompt Inversion), and maintains the diffusion model's editability (compared EDICT).
Are such optimizations indispensable and optimal for diffusion-based image editing?
Optimization-basae inversion methods learn specific latent variable to minimize the loss function z -/-0, Thus, the target of optimization-base inversion is to correct 2, learned latent variable then serves an input for both the source and target branches.
F.r back to =t" The
The optimization of a unified variable for source and target branches leads to several problems: (1) To optimize the specific latent variable, prolonged processing time is needed during inference (e.g., 148.48 seconds per image for Null-Text Inversion), which is impractical for editing tasks with user interaction; (2) Considering that extended optimization times not expected, previous approaches have opted to optimize the target latent for only a limited number of iterations. Consequently, the result frequently entails a learned latent space with a discernible gap between = and the initial =00h especially when a large distance exists between 2 F.x and =i" This leads to a decline in essential content preservation ability, as shown in our ablation study; (3) The learned variable serves as the generation model's input parameter, which is not aligned with the diffusion model's expected input distribution and leads to negative impacts on the diffusion model integrity and 2," fh These three issues hinder the practicality and editability of these optimization-bar based inversion methods.
4.2 METHOD
Bearing these issues into consideration, we propose PnP Inversion. The key of PnP Inversion is to disentangle the source and target branch, thus enabling each branch to unleash its maximum potential individually. In the source branch, we can directly add 2,"" 2, Fng back to F.,, which is a simple strategy that can directly rectify the deviation path and is plug-and-play to various editing methods. In the target branch, simply leaving it unaltered would maximize the diffusion models' potential for target image generation. This simple but effective solution solves the three issues in optimization- based inversion by (1) No optimization required, thus incurring minimal additional time overhead: (2) Adding 2 2t P.g climinates the discernible gap between 2 and the initial =0 rrc (3) Do not have any impact on the distribution of the diffusion model's input and at
Pco
Algorithm D presents the algorithm for inserting PnP Inversion into existing diffusion-based image editing methods. Red lines with gray backgrounds highlight the 3 lines of code added by PnP Inversion. Typical diffusion-basee editing of images involves two parts: an inversion process to get the diffusion space of the image, and forward process perform editing on the diffusion space.
Specifically, given a source image or latent embedding 20"o, we first use DDIM Inversion with source prompt O to obtain the perturbed diffusion space 2 in Part I (Algorithm I line 1-5). Then, in Part (Algorithm [ line 6-8), existing editing methods take in source prompt Core and target prompt input with a batch size of 2 to perform essential content preservation and editing by integrating information between two batches. We use [] to enclose the elements in a batch for better illustration. E.g., [a,b] represents a batch with and b concatenate in batch dimension.
Chea simultaneously
PnP Inversion can be plug-and and-el into the forward process in Part II and rectifies the deviation path step by step as shown in Algorithm i line 9-13. Particularly, line 10 computes the perturbed
5

---

Published as a conference paper at ICLR 2024
Then, df the difference between the perturbed DDIM Inversion latent z and 2t-1 the DDIM forward latent with C conditions, is calculated in line 11. Finally, in line 12, we add the dr and only to the source/reconnsrrucion branch in DDIM Forward Editing Madel, which is the key to rectifying the deviation path as well as retaining the editability of the target prompt's latent space.
forward latent Fig of the source branch in DDIM forward as shown in Figure Fag
2(-
Algorithm 1: Real Image Editing Pipeline with PnP Inversion
Input: A source prompt embedding Care (or embedding of null for some editing methods), a target prompt embedding C* a real image or latent embedding =6"" a prompt-hased image editing method DDIM_ [orward Editing-Moded (e.g., Prompt-to-Promy
Output: A reconstruction image latent embedding Faa
edited image latent embedking Fts
Part I Invert =
20
Amc. 20
for
T 1,7 do
DDIMMInversin ([=E)) , 1, 1.(orr));
end
Pag PT
Pig 2T
/0 Part II: Perform editing on Zyy for
-T.T
I...., do
Fog Fig 11-1: F1-1
DDIM Forward Model ([ , 200r] ,t, [C "0rr, CC
end
Part II: Perform editing on 2T I
with PnP Inversion
for f
T,T
1 do
DDIM Forward
C
11
//
2
12
Pag Fig 7t-11 f1-1122-1
DDIM Forward Editing Model (1 12 200)
1Cure, Ctat
d dree
//
13 end
14 Return Fen
4.3 BENCHMARK CONSTRUCTION
Editing Type
Source Image
Source Prompt
Target
Editing Instruction
Editing Mask
Editing Type
Source Image
Source
Target
Editing Instruction
Editing Mask
Prompt
Prompt
Prompt
(doa)
[cae] stann
Change from doa
(walking]
preyouudd
greyhound
[jumptng]
Chazge bray
mirouhh
through
chair
coess
grass
Junping
Change the
Change Object
Pisthes] the cosan
[Sharkss] in the oocan
Change the lishes to sharkss
Change Color
[witito]] stitt
[yellow]
stirt
color of the
start from white
yellow
Asan woman Elowers
[Wooden
Object
woman's
Change
polaroid,
photogmphy
byy polamit, travel photography
Make cansera weoden
hair
Delete Objjctt
A painting Lan orange chair and] lump in Tiving momm
lamp in Tiving ng momm
painting of
Remove The chair
Change Backrgound
girl sitting the [ruins]
A girl sltting la the [baach]
Change the backgmand from rulns besch
Change
[Painting ofl [A photo bird
Change
Change
Esgurkding]
31e0s
traes [blood]
speklingg to
Change
standingg on standing on tree
image from illusration
river
tree beanch
photo
Figure 3: Examples of PIE-Bench. One example is provided for each editing type. For each
sample, we have provided corresponding annotations of editing type, source image, source prompt, target prompt, editing instruction, and editing mask (the mask region is in yellow).
While diffusion- based editing has garnered significant attention in recent years, evaluations of var- ious editing methods have primarily relied on subjective and limited visualizations. To systemat- ically validate our proposed method as a plug-and-pla strategy for editing models and compare method with existing inversion methods, well as compensate for the absence of standardized

---

Published as a conference paper at ICLR 2024
performance criteria for inversion and editing techniques, we construct benchmark dataset, named PIE-Bench (Prompt-based Image Editing Benchmark).
PIE-Bench comprises 700 images featuring 10 distinct editing types. Images are evenly distributed in natural and artificial scenes (e.g., paintings) among four categories: animal, human, indoor, and outdoor. Each image in PlE-Bench includes five annotations: source image prompt, target image prompt, editing instruction, main editing body, and the editing mask. Notably, the editing mask annotation (indicating the anticipated editing region) is crucial in accurate metrics computations as we expect the editing to only occur within a designated area. Details are in the supplementary files.
5 EXPERIMENTS
Due to page limitation, we only provide the comparison of inversion-based editing, essential con- tent preservation methods, ablation on PnP Inversion and Null-Text Inversion, and the influence of adding the difference to target latent in this section. More experiments are in supplementary files.
5.1 EVALUATION METRICS
To illustrate the effectiveness and efficiency of our proposed PnP Inversion, we use eight metrics covering four aspects: structure distance (Tumanyan et al. (222). background preservation (PSNR, LPIPS (Zhang al. al.] 2018), MSE, and SSIM (Wang et al. al.] (204)) outside the annotated editing mask), edit prompt-image consistency (CLIPSIM (Wu et al. (221)) of the whole image and regions in the editing mask) and inference time. Details can be found in the supplementary files.
5.2
COMPARISON WITH INVERSION-BASED EDITING
Method
Structure
Background Preservation
CLIP Similariy
Inverse
Editing
Distance 1
PSNR
LPIPS, 100 MSE LSSMM k t|
Whole
Edited f
DDIM
P2P
69.43
17.87
208.80
219.88
71.14
25.01
22.44
NT
P2P
13.44
27.03
60.67
35.86
84.11
24.75
21.86
NP
P2P
16.17
26.2T
69.0T
39.73
83.40
24.61
21.87
StyleD
P2P
11.65
26.05
66.10
38.63
83,42
24.78
21.72
Ours
P2P
11.65
(27.22
54.5574%
32.86.5%J
84.76.9%
[25.02
22.10, .766
DDIM MasaCtrl|
28.38
| 22.17
106.62
86.97
79.67
23.96
21.16
Ours MasaCtrl|
24.70
222.642%1
17.94
81.097%.
81.33
[24.38|
21.35
DDIM P2P-Zero
61.68
20.44
172.22
144.12
74.67
22.80
20.54
Ours P2P-7ero
49.2220%
| 211553
138.9819%..
127.32,2%
77.05
[23.31 22..31..2%1
21.05 2
DDIM
PnP
28.22
22.28
113.46
83.64
79.05
25.41
22.55
Ours
PnP
14%1
22.46.%
106.067%4
80.45
4%6
79.68,
25.41
22.62
use Stable Diffusion v1.5 as base model (others all use Stable Diffusion v1.4)
averraged results A800 and RTX3090 since different environment leads to slightly different performance
Table 1: Compare PnP Inversion with other inversion techniques across various editing meth-
ods. For editing method Prompt-to-Prone
(P2P) (Hertz et al. (2023). we compare four different in-
version methods: DDIM Inversion (DDIM) (Song et al.] 2020). Null-Text Inversion (NT) (Mokady
et al. 2023), Regative-Prompl Inversion (NP) (Miyake et al. 2023), and StyleDiffusion (StyleD)
al.
2023b). For editing methods MasaCtul (Cao et al. (023). Pix2Pix-Zon (P2P-Zero) (Cao al.
2023
Plug-and-Play (PnP) (Tumanyan et al. 2023). we compare with DDIM Inversion (DDIM).
This section compares PnP Inversion with previous inversion-based edit- ing methods quantitatively and qualitatively. Four inversion methods, DDIM Inversion (Song et al., 2020).. Null-Text Inversion (Mokady et al.)
Method Time
NP
18.22
EF
19.10
2023). Negative-Pronee Prompt Inversion (Miyake et al.] 2023), and StyleDif- fusion (Li et al. 2023b), as well as four editing methods, Prompt-to- Prompt (Hertz et al. 2023), MasaCtrl (Cao et al.||202) pix2pix-eerx (Par-) mar et al. 2023), 3), and Plug-and-Play (Tumanyan et al. (203)) are taken into consideration. For inversion methods such as EDICT (Wallace et al. 2023)
AIDI
35.41
EDICT
35.48
NT
148.48
StyleD
382.98
Ours
28.17
and Edit Edii-FFrendyy DDPM Inversion Hluberman-Spieger we put them into section 5.3 since their main target is to preserve the back- ground and result in decline in editability.
et al. 2023).
Table 2: Comparison of inference time.
7

---

Published as a conference paper at ICLR 2024
round cake with orange frasting wooden plate square cake with orange frusting wooden plate
A drawing of man with blue eyer A drawing of a robot with blue eyes
A gold plated bowl filled with fruit - A plated plated fille
filled with candy
A woman silting living
A watercolar painting of ritting fiving
(R)
(h)
Source Image DDIM + P2P
NT P2P
NP P2P
StyleD + P2P Instruct-PPP Blended-L-D Ours P2P
Figure 4: Visualization results of different inversion and editing techniques. The source image is shown in col (a). We compare (h) PnP Inversion with different inversion techniques added with
-or-mtpp (Hertz et al., 2023): (b) DDIM Inversion (Song al. 2020), (c) Null-Text Inver- Inversion (Miyake et al. 2023), and (e) StyleDiffu- sion| et al. 2023b). We also compare model-based editing results: (f) Instruct-Pix2Pix (Brooks
sion Mooaddy et al. (023), (d) Negative-
al. 2023) and (g) Blended Latent Diffusion (Avrahami et al. (002)) The improvements are mostly tangible, and we circle some of the subtle discrepancies w/o PnP Inversion in red.
Table [ shows the structure retention, background preservation, and edit CLIP Similarity of the four inversion methods and the four editing methods. Results show that when added with PnP Inversion, all editing methods have a performance improvement on the retention of background and structure while improving or maintaining editability compared with other inversion methods. While DDIM Inversion may yield a high edit CLIP Similarity within the edited mask, the preservation of structure and background falls significantly below the acceptable performance threshold, depicted in Figure 4 We also give out the inference time of Srggative-Promp Inversion (NP), Edit Friendly Inversion (EF), AIDI (Pan et al.) 2023). EDICT, Null-Text Inversion (NT), and Style Diffusion (SD) added with Trompt-to-Promp in Table 2 PnP Inversion achieves better editing results with far less inference time than Null Text Inversion and StyleDiffusion. Although Negative-Prome
Inversion and Edit Friendly DDPM infer little faster than PnP Inversion, their editing results are much more unacceptable compared to PnP Inversion as shown in Table I and Table
5.3 COMPARISON WITH ESSENTIAL CONTENT PRESERVATION METHODS
We compare PnP Inversion with inversion and editing techniques targeted for essential content preservation in Table 3 Null-Text In- version (NT) (Mokady al. 2023) added with Prompt-to-Prome (P2P) (Hertz et al.] (023) provides a baseline for all improvement methods. Specifically, Negative-Prompt Inversion
Method
CLIP Similariy
Inverse Edit
Whole 1 Edit f
NT
P2P
24.75
21.86
(NP) (Miyake et al., 2023) maintains a guidance scale of to reduce the devia- tion in editing. Proximal Guidance (Prox) (Han al., 2023) lim- its edit changes to a specific area based on editing amplitude. Edit Friendly DDPM (EF) (HuberrmnnSpiieellllaa et al. 2003) changes the DDPM sampling distribution to allow reconstruction of the de- sired image. EDICT (Wallace et al.] (023) maintains two coupled noise vectors to invert each other for image reconstruction. Al- though some of these techniques improve the structure and back- ground preservation compared to Null-Text Inversion, clip similarity has decreased for all meth- ods, which indicates a deteriorating editing ability. On the contrary, PnP Inversion can lift struc- ture/background preservation and editability simultaneously, shown in Table 1
NT
Prox
22.91
|20.23]
NP
Prox
24.28
21.36]
EF
P2P
23.97
21.03]
EDICT
P2P
23.09.
20.32
Ours
P2P
25.021
122.101
Table 3: Compare PnP
Inversion with background preservation methods.
8

---

Published as a conference paper at ICLR 2024
5.4
ABLATION STUDY
5.4.1 COMPARING PnP Inversion AND NULL-TEXT INVERSION
To validate our theoretical analysis, we prove experimentally in Table that PnP Inversion's 's im- provement over Null-Text Inversion (NT) is a three-step process, disentangling the source and target branch, wiping off the force assignment of null-text embedding, and removing the distance gap shown in Figure 2 To disentangle the two branches, we revise Null-Text Inversion to a single- branch version (NT-S) and only assign the learned null-text latent to the source branch. Results show an improvement in CLIP Similarity, revealing the benefit of leaving the target branch unal- tered. To wipe off the force assignment, we use the optimization strategy of Null-Text Inversion, and instead of replacing null-text embedding, we directly add the difference to the source latent. The result is shown Null-Latent Inversion (NL). To show the influence of the distance gap, we add scaled distance (scale * df to the source latent. Results show that with the distance gap increase, the structure and background preservation decline, while the edit fidelity fluctuates. Moreover, the Null-Latent Inversion's s performance is between added distance with a scale of 0,4 and 0,8, which implies the average optimization distance gap Null-Text inversion between 0.4 and 0.8.
Metries
Structure
Background Preservation
CLIP Similarity
Method
Distance
x10=
PSNR f
LPIPS x + MSE +
SSIM x102
Whole f
Edited
NT NT-S*
13.44
27.03
60.67
35.86
84.11
24.75
21.86
14.25t
26.39.
66.621
40.091
83,52]
25.011
22.111
Scale
Distance x103
PSNR t LPIPS x103 L MSEx104 & SSIM2100
Whole f
Edited
0.4
13.55
26,65
58.79
36.98
84,29
25.02
22.10
NL
12.05L
27.03*
55.83
33.94
84.551
25.021
22,09t
0.8
11.90
27.14
54.76
33.35
84.66
25.08
22.12
11.65
27.22
54.55
32.86
84.761
25.021
22.101
averaged results A800 and RTX3090 since different environment leads slightly different performance
Table 4: Ablation study of comparing Null-Tex Inversion and PnP Inversion.
5.4.2 INFLUENCE OF ADDING DIFFERENCE TO TARGET LATENT
In Algorithm|] 1 we only add the distance of the source prompt to the source latent. To show the ra- tionality this operation and the disentanglement of the source and target branch, we compare the performance of adding source/reconstruction distance di to the target latent and adding target dis- tance d,2h, to the target latent in Table where the operation on source branch is leaved unchanged. Specifically, d'th is computed by changing Crore to C'us in Algorithm line 10. Adding source distance the target latent leads decline in both structure/background preservation and clip similarity. Although adding target distance to the target latent leads to better structureebbcckgroun preservation, the clip similarity (edit fidelity) sharply decreases.
Metries Structure
Background Preservation
CCLPP Similariy
Add "Distancex.o> J PSNR L.IPS x 4 MSE 4. SSIM 1p
Whole Edit t
drec
19.30
26.15
63.70
46.45
83.67
24.93
21.88
dige
9.86
27.66
50.17
33.28
85.13
23.00
20.27
11.65
27.22
54.55
32.86
84.76
25.02
22.10
Table 5: Results of adding the difference to the target latent.
CONCLUSION
This paper introduces PnP Inversion, a simple yet effective technique for inverting diffusion models. By disentangling the source and target branches in diffusion- editing, PnP Inversion separates the responsibility for essential content preservation and edit fidelity, thus achieving superior perfor- mance in both aspects. To address the lack of standardized performance criteria for inversion and comprising 700 images in natural and artificial scenes editing techniques, we develop Evaluation metrics demonstrate that PnP Invession outperforms featuring ten distinct editing types. eight editing methods across five inversion techniques in terms of both edit quality and inference speed. Limitations and future work can be found in supplementary files.

---

Published as a conference paper at ICLR 2024
Acknowledgment. This work supported in part by the Innovation and Technology Fund under Grant No. MRP/022/20X, and part by Research Matching Grant (CSE-7-2022) RMG01.
REFERENCES
Omni Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18208-18218, 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 1211) 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble expert denoisers. arXiv preprint arXiv:2211.0132. 2022.
Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprint arXiv:2301. 12247, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2023.
Recognition (CVPR), pp.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning- free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08444 2023.
Ling-Hao Chen, Jei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia, and Tongliang Liu. Human- mac: Masked motion completion for human motion prediction. Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pp. 9544-9555, October 2023a.
Ling- Ling-Hao Chen, Yuanshuo Zhang, Taohua Huang, Liangcai Su, Zeyi Lin, Xi Xiao, Xiaobo Xia, and Tongliang Liu. Erase: Error-resilient representation learning graphs for label noise tolerance. arXiv preprint wxXiv:2322.08852, 2023b.
Bin Cheng, Zuhao Liu, Yunbo Peng, and Yue Lin. General image-to-image translation with one-shot image guidance. arXiv preprint arXiv:2307.14352 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In International Conference Learning Representations (ICLR), 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34:8780-8799 2021.
Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text-driven image editing using diffusion models. arXiv preprint arXiv:2305.0444// 2023.
Adham Elarabawy, Harish Kamath, and Samuel Denton. Direct inversion: Optimization-fis -free text- driven real image editing with diffusion models. arXiv preprint arXiv:2211.00 1.07825, 2022.
Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-free textual inversion. arXiv preprint arXiv:2304.05878, 2023.
Weixi Feng. Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for 2022.
compositional text-to-image synthesis. arXiv preprint
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 22X. 2022.
10

---

Published as a conference paper at ICLR 2024
Zigang Geng, Binxin Yang. Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al. Instructdiffusion: A generalist modeling interface for vision tasks. arXiv preprint arXiv:2309.03880 2023.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu 0003, Qilong Zhangli, al. Improving tuning- free real image editing with proximal guidance. CoRR, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Trompt-to-prompn image editing with cross-attention control. In International Conference Learning Representations (ICLR), 2023.
Jonathan Ho and Tim Salimans. arXiv:2207.12599 2022.
Classifier-free diffusion guidance.
arXiv preprint
Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu, Reversion: Diffusion- based relation inversion from images. arXiv preprint arXiv:2303. 13495, 2023.
Inbar Snseerman-Spiegelllaa Vladimir Kulikov, and Tomer Michaeli, An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint urXiv:2304.06140, 2023.
KJ Joseph, Prateksha Udhayanan, Tripti Shukla, Aishwarya Agarwal, Srikrishna Karanam, Koustava Goswami, and Balaji Vasan Srinivasan. Iterative multi-granular image editing using diffusion models. arXiv preprint arXiv:2309.0000 00010,,,, 2023.
Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: A native skeleton-guidet diffusion model for human image generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15988-15998, 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6007-6017, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2426-2435, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 2023a.
Semmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing. arXiv preprint arXiv:2303. 15649, 2023b.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.0477 047611 2023a.
Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 10462-10472. 2022a.
Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-awee aware implicit neural audio-driven video portrait generation. In European Conference on Computer Vision, pp. 105-125. Springer, 2022b.
Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: Hyper-realisee human generation with latent structural diffusion. arXiv preprint arXiv:23 2023b.

---

Published as a conference paper at ICLR 2024
Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprint arXiv:2311.17061, 2023c.
Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung- Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310. 129778 2023.
Yue Ma, Xiaodong Cun, Yingqing He, Chenyang Qi, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Magicstick: Controllable video editing via control handle transformations. arXiv preprint arXiv:2312. 12.00477 2023a.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Fol- low your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01 186, 2023b.
Chenlin Meng, Yutong He, Yang Song. Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. Interna- fional Conference on Learning Representations (ICLR), 2022.
Ashkan Mirzaci, Tristan Aumentado- Armatron, Marcus A Brubaker, Jonathan Kelly, Alex Levin- shtein, Konstantinos G Derpanis, and Igor Gilitschenski. Watch your steps: Local image and scene editing by text instructions. arXiv preprins arXiv:2308.0894,, 2023.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompp inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint arXiv:2305. 16807, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8038-6047 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning (ICML), pp. 16784-16804. PMLR, 2022.
OpenAl. Gpt-4 technical report.
ArXiv, abs/2303.08774, 2023.
URL https://api.
semanticsse Semanticscholar.
Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accelerated iterative diffusion inversion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15912-15921, 2023.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yar Zhu. Zero-shot image-to-imagee translation. In Special Interest Group Computer Graphics and In- teractive Techniques (SIGGRAPFI), pp. 1-11, 2023.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint 2003::2303.09535.0 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML), pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204. 06/25, 2022.
12

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022.
Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdif- fusion: Harnessing diffusion models for interactive point-based image editing, arXiv preprint arXiv:2306. 14435, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations (ICLR), 2020.
Linoy Tsaban and Apolinário Passos. LEDITS: Real image editing with ddpm inversion and seman- tic guidance. arXiv preprint arXiv:2307 00522, 2023.
Narek Tumanyan, Omer Bar- Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10748-10757, 2022.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 1921-1930, 2023.
Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image edit- ing by fine tuning an image generation model on single image. arXiv preprint 200002000077 2022.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled trans- formations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 22532-22541, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4))600- 612, 2004.
Chen Henry Wu and Fernando De la Tore. Unifying diffusion models" latent space, with applica- tions to cyclediffusion and guidance. arXiv preprint arXiv:2210.055550 2022.
Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero- shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7378-7387, 2023.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. GODIVA: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Pp. 1900-1910, 2023.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 45(3):121-31338 2022.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effecctivnness of deep features as perceptual metric. In Proceedings of the IEEE/CVF CGnf/eveCoonfeeence on Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018
Yuechen Zhang, Jinbo Xing, Eric Lo, and Jiaya Jia. Real-world image variation by aligning diffusion inversion chain. arXiv preprint 2r3Xiv2305.110 18729, 2023a.
Zhixing Zhang, Ligong Han, Amab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6027-6037, 2023b.
13

---

Published as a conference paper at ICLR 2024
Reproducibility Statement. To ensure the reproducibility and completeness of this paper, we in- clude the Appendix with sections. Appendix A provides details of related works, offering ad- ditional information to complement the main text. Appendix B introduces the construction of PIE-Bench in detail and provides examples in the benchmark. Appendix C illustrates the evalu- in experiments. Appendix (eanndxx(ccoooe conttiiss the details of implementation. Appendix E contains quantitative results on the reconstruction ability of different inversion meth- ods, full comparison with essential content preservation methods, a comparison with model-based editing, ablation of step and interval, influence of inverse and forward guidance scale, and results of different editing types. Appendix [ provides more qualitative results compared with different inversion-based editing, essential content preservation methods, and model-based editing. Lastly, we include limitations and future works Section
ation metrics
A RELATED WORK
Diffusion models excel in multiple generation tasks (Liu al. 2022b a 2023c Ju al. 2023 Ma et al. 2023a Chen et al. 2023a by Lu et al.) (023). As mentioned in the main paper, diffusion- based image editing involves two primary concerns: (1) edit fidelity and (2) essential content preser- vation. Most diffusson-based editing methods take both aspects into consideration and perform editing using two branches strategy, ie., a source diffusion branch to maintain the source im- age's essential content and target diffusion branch insert editing instruction (shown in Figure Accordingly, we have a comprehensive review of prior methodologies concerning both two aspects.
Optimize
Target Branch Editing
Deflected
Imnge
End-to-end Model
Latent Integration
Attention Integration
Target Embedding
Artention Map
DDIM
Edited Image
Inversion
Target Procept: doy
Source Prompt: A cat
Source Branch Preservation
Map
Single Image
DDPM Variation
Source Image
Attention Preservation
Embedding
Optimize
Diffusion-based editing pipeline showing how edit fidelity and essential content preser- Figure 5:
vation are achieved in different methods. Detailed introduction is in supplementary files.
Edit Fidelity. Diffusion models naturally possess hierarchical features (e.g., noisy latent of each step, different resolution UNet features), enabling different editing strategies. Previous methods perform editing roughly through four ways: @ end-to-end editing model (with only one editing branch), @ latent integration, @ attention integration, and target embedding.
Specifically, o trains end-to-end diffusion models for image editing, which is limited by insuffi cient/noisy training data or indirect training strategies (Brooks et al. 2023 Kim et al. 2022 Nichol al. 2022 Mirzaei et al. 2023). The shared objective of (2-4 is map both the source image and the target editing instruction to the diffusion space, then inject the target branch's features into the souree diffusion space. @ inserts editing instruction in the level of the noisy diffusion latent (Meng
et al. 2022, Avrahami et al. 2022 2023 Couairon et al. 2023) Zhang et al.) 2023b] Joseph et al.
2023
Elarabawy al. 2022 The features of the source and the target branch merged through
14

---

Published as a conference paper at ICLR 2024
Couairon et al. 2023 Joseph al, 2023) or weighted addition (Zhang et al. 2023b Shi et al. 2023). However, using mask stitching for feature insertion may lead to abrupt editing boundaries, and using weighted addition makes difficult to make refined modifications. @ tries to solve these two problems by fusing in a more refined feature space, the attention map that connects the text and image. Prompt-to-prompt (Hertz itt al., 2023) directly replaces the cross-attention map to perform editing through text. Proximal guidance (Han al. ).22), Zero-shot (Parmar al. )023), , MasaCtrl (Cao al., 2023). Plug-and- Play (Tumanyan et al. (023). RIVAL (Zhang et al. 2023a), and DragonDifiusion (Mou et al., 2023) further extend the use both cross-attention and self-attention map achieve better editing results or explore more applications. @ first aggregates editing information into embedding, then uses this embedding to perform editing on source diffusion branch, which may confront long feature extraction times and unstable editing performance (Kawar et al.
mask stitching (Meng al., 2022 Avrahami et al. 2022 2023
2023
Cheng al. 2023 Wu et al.
2023
Brack et al. 2023
Tsaban & Passos 2023
Valevski et al.
2022 Dong et al. 2023).
Source Branch
Target Branch
Source Branch
Target Branch
Method
Method
(
1
Mokady et al. 2023
Cao et al.
2023
Mou 2023
Brooks et
2203
al. 2022
Nichol
2022
Mirzaci (2033
Meng ct al.
2022
Avrahami ct al 2023
Avrahami ct (202
Couatron al. 202.3
Zhang et al.
202.3b
Shi et al. 202.3
Hertz et al.
2125
(02.3)
Parmar et a
Tumanyan of al. (2023
Zhang et al.
2023a
Kawar et al. (2023)
Cheng et al.
2023
Wu et al. (2025)
BBrakk
202.5
FTTTA
Tsahan Passos 2023
Valevski
Dong ct al. (
Miyake et al. 21)2.
Huberman-Spicge glas et al.
2023
Qi ct al. 20223
Li al. 20235
Gal et al.
(2022)
Fei al. 2025
Huang 2023
Wallace al. [2023
Joseph et al.
2023
Geng of al.
2023
Table 6: Strategies for enhancing editing fidelity and preserving essential content in previous
diffusion- editing methods.
Essential Content Preservation. While the methods mentioned enable basic image editing, pre- serving the essential content, particularly images devoid of inherent diffusion space, remains challenging. Previous methods tried to solve this problem through overfit the editing image, ? DDPM/DDIM inversion variation, ? attention preservation, and O source embedding.
Specifically, overfits the source image to avoid significant image variation (Kawar et al. 2023 (hi et al. 2023) makes variations the DDPM/DDIM sampling process to adapt the edit- ing. Negative-prompt Inversion (Miyake et al. 2023) set the classifier-free guidance scale to reduce the deviation caused by DDIM inversion, which weakens the text's controllability. Edit et al.
Friendly Noise (Fuueeraan-Spieeelgall the noise space to ensure better reconstruction. However, this reduces the modification space due to the decrease in noise. EDICT (Wallace et al.) 2023) maintains two coupled noise vectors to reach mathematically exact inversion, leading to a decrease of edit fidelity. O devises ways utiliz- ing both rross-attention and self-attention map with better balance of semantic editing results and original image structure (Mou et al.
2023) imprints the source image more strongly onto
2023
Cheng et al. 2023 Cao et al. 2023 Parmar et al. 2023 Qi et al. 2023). 3)
Tumanyan et al.
2023 Hertz et al.
2023
4 absorbs source image an embed-
ding and use this embedding to reconstruct the essential content of the source image (Mokady et al.
2023 Dong al.
2023
Li al. 2023b; Gal et al. 2022 Fei et al. 2023 Huang al. (023).
Specifically, Null-text inversion (Huang et al.) (203)) optimizes a Null embedding to capture the difference between the reconstructed image and the source image. Subsequently, this difference is
15

---

Published as a conference paper at ICLR 2024
steply reintroduced in both source and target branch during the editing procedure. However, null- text inversion necessitates prolonged optimization times per image, lacks the assurance of achieving flawless optimization, and disturbs the diffusion model distribution. Prompt Tuning Inversion (Dong et al. 2023) and StyleDiffusion (Li et al. 2023b optimize text embedding and
value capture the difference instead of null-text, thus facing the same issue with Null-text Inversion.
More refined categorization is presented in Tables6 To summarize, existing background preservation methods suffer from unstable and time-consuming optimization processes, as well as the persisting issue of errоr propagation inversion. Moreover, the absence of a disentanglement for the source and target branches is unfavorable for achieving optimal performance in both edit fidelity and essential content preservation. Instead, simple yet effective PnP Inversion is capable of achieving superior results with virtually negligible computational cost and negligible inversion error without optimiza- tion by branch disentanglement, aiding in accurately editing the real images while preserving the structural information.
B BENCHMARK CONSTRUCTION
Although diffusion-based editing has been widely explored in recent years, people mainly evalu- ate the performance of different editing methods with subjective and incomprehensive visualiza- tion results. Previously, PnP (Tumanyan et al.) (023) provides benchmark of 55 images with editing prompts. Instruct-Pix2Pix (Brooks et al.) 2023) builds dataset with randomly selected 451,990 images, editing prompts written by ChatGPT, and pseudo editing results of Null-Text In- version (Mokady et al., (023) and Prompt2Prompt (Hertz et al., (023). However, without manual labels and fine-grained classification, these datasets cannot support comprehensive metrics evalua- tion.
To systematically validate our proposed method as plug-and-play strategy for editing models and compare our method with existing inversion methods, as well as compensate for the absence of standardized performance criteria for inversion and editing techniques, we construct a benchmark dataset, named PIE-Bench (Prompt-based Image Editing Benchmark).
PIE-Bench comprises 700 images in natural and artificial scenes (e.g., paintings) featuring ten dis- tinct editing types shown in Figure 3] (0) random editing written by volunteers (140 images), (1) change object (80 images), (2) add object (80 images), (3) delete object (80 images), (4) change object content (40 images), (5) change object pose (40 images), (6) change object color (40 im- ages). (7) change object material (40 images), (8) change background (80 images), and (9) change image style (80 images). In each editing type of 1-9, images are evenly distributed among natu- ral and artificial scenes. Within each scene, images are evenly distributed among four categories: animal, human, indoor environment, and outdoor environment. Each image in PIE-Bench includes five annotations: source image prompt, target image prompt, an editing instruction, edit sub- jects describing the main editing body, and the editing mask. For editing type O, we invited some volunteers to write the source image prompt, target image prompt, and editing instructions based on their editing expectations. For the other editing types, we employ BLIP-2 ( et al., 2023a) to generate the source image prompt and use GPT4 (OpenAl 2023) to craft the target image prompt and editing instructions tailored to each editing type. Then, we manually rectify any inaccuracies in the automatically generated captions, target prompt, and edit instructions. Subsequently, 2 data annotators and data auditor annotate the main editing body as well as the editing mask (indicat- ing the anticipated editing region) in an image. Notably, the editing mask annotation is crucial in accurate metrics computations as we expect the editing to only occur within the designated area.
C EVALUAION METRICS
To illustrate the effectiveness and efficiency of our proposed PnP Inversion, we use eight metrics covering four aspects: structure distance, background preservation, edit prompt-image consistency and inference time.
Structure Distance: We follow Tumanyan et al. (2022) leverage self-similarity of deep spatial features extracted from DINO- VIT as a structure representation and use cosine similarity between image features as structure distance. The structure distance can capture structure while ignoring
16

---

Published as a conference paper at ICLR 2024
appearance information. Thus. it is well-suited for our proposed benchmark and diffusion-basee editing methods since we do not expect a huge structural change.
Background Preservation: We calculate standard PSNR, LPIPS (hhagg al. 2018). MSE, and SSIM (Wang et al. 2004) the area outside the manual-annotatee annotated masks of PIE- PlncBench to demon- strate the background preservation ability of different inversion and editing techniques.
Edit Text-image Consistency: The CLIP (Rafford et al, 2021) Similarity (CLIPSIM (Wu et al,
2021 evaluates text-image consistency between the edited images and corresponding target editing text prompts. CLIP Similarity projects text and images to the same shared space and evaluates the similarity of their embeddings. We calculate CLIP Similarity both on the whole image and in the editing mask (black out everything outside the mask) to demonstrate the performance of editing, as well as reflecting the editability. These two metries called Whole Image Clip Similarity and Edit Region Clip Similarity, respectively.
We inference time per image of different inversion techniques and Prompt-to- Prompt (Hertz et al.) (023) on one NVIDIA A800 80G to evaluate efficiency. Results are averaged over random runs.
Inference Time:
D IMPLEMENTATION DETAILS
We perform the inference different editing and inversion methods in the same setting unless specifically clarified, i.e., on RTX3090 following their open-source code with base model of Stabe Diffusion v1.4 50 steps, with an inverse guidance scale of and forward guidance scale of O. Different images may have different hyper-parameters in different editing models, and we keep the recommended hyper-parameter for each editing method in all images for fair comparison. Details can be found in the provided code.
E QUANTITATIVE RESULTS
E. RECONSTRUCTION ABILITY OF DIFFERENT INVERSION METHODS
To further show the reconstruction ability of different inversion methods, evaluate the recon- struction results of DDIM Inversion, Null-Text Inversion, Negative-Prompt Inversion, StyleDiffu- sion, and PnP Inversion by giving source prompt model input. We provide results of Structure Distance and Background preservation show the ability to correct 2 back to zi. As shown in Table 7 PnP Inversion is better than all these inversion methods on all metrics.
Inverse
Structure Distance, x
PSNR
LPIPS x10" 4
MSE +
SSIM x102 1
DDIM
70.23
17.76
210.84
224.43
70,96
NT;
3.30
30.17
33.39
18.86
86,84
NP
8.47
27.73
57.04
30.05
84.59
StyleD
4.35
28.88
39.45
22.63
86.07
Ours
2.95
30.57
31.41
17.60
87.20
averaged results A800 and TRX3090 since different environment leads slightly different performance
Table 7: Reconstruction results of different inversion techniques
E.2 COMPARISON WITH ESSENTIAL CONTENT PRESERVATION METHODS
We provide the full table comparing PnP Inversion with inversion and editing techniques targeted for background preservation in Table g As explained in the main paper, although some these techniques improve the structure and background preservation compared to Null-Text Inversion, clip similarity has decreased for Proximal Guidance (Prox), Prgative-Prompe Inversion (NP), Edit Friendly Inversion, and EDICT, which indicates deteriorating editing ability. However, PnP In- version can lift structure/backgroum. preservation and editability simultancously, which shows the effectiveness of PnP Inversion in image editing.
17

---

Published as a conference paper at ICLR 2024
Method
Structure
Background Preservation
CLIP Similariy
Inverse
Editing Distance 108
|PSNR LPIPS 1 MSE SSIM, x 10
Whole
Edit f
NT
P2P
13.44
27.03
60.67
35.86
84.11
24.75
21.86
NT
Prox
3.51
30.211
32.97
18.47_
87.01
22.91
20.23]
NP
Prox
7.44
28.671
41.98
24.25
85.981
24.28
21.36]
EF
P2P
18.051
24.55L
91.88t
94.581
81.57
23.97.
21.03]
EDICT
P2P
4.61
29.797
37.03
20.37
86.551
23.09,
20.32,
EDICT
13.28
26.76L
65.511
38.14
83.72
24.46
21.56]
AIDI
P2P
12.194
26.96
57.921
39.82,,
84.171
24.967
22.01
Ours
P2P
11,65.
27.221
54,55
32.86,
84.761
25.021
22.101
OurS+AIDI
P2P
11.54.
27.261
54.54.)
32.78..
84.691
25.02
22.091
Table Full table comparing PnP Inversion with background preservation methods. Null- Text Inversion (NT) (Mokady al.) 2023) added with Prompt-to-Prompt (P2P) (Hertz et al.)
2023) provides a baseline for all improvement methods. Specifically, Srgaatvee-Promp Inversion
(NP) (Miyake al. 2023) maintains guidance scale of reduce the deviation in editing. Proxi- mal Guidance (Prox) (Han et al. (023) limits edit changes to a specific area based on editing ampli- tude. Edit Friendly DDPM (EF) (Huberman-SSiegeelgaane
al. 2023) changes the DDPM sampling
distribution to allow reconstruction
the desired image. EDICT (Wallace et al.) (023) maintains
two coupled noise vectors to invert each other for image reconstruction. AIDI (Pan al. (023)
employs an iterative procedure to find a fixed-point solution for the ideal diffusion latent.
Accelerated Iterative Diffusion Inversion (AIDI) Pan et al. (2023) is also an essential content preser- vation approach that employs iterative procedure to find fixed-point solution for the ideal diffu- sion latent, as depicted by the dashed circle in Figure 2. AIDI is distinct from and complementary our approach; while AIDI concentrates at the DDIM Forward correction. We have conducted experiments to both compare and combine our method with AIDI. The table above illustrates that our method outperforms AIDI in terms of structural distance, background preservation, and editability. Additionally, the integration of AIDI into our method results in further enhancements to performance. This improvement substantiates the orthogonality and potential synergistic relationship between our approach and AIDI.
refining DDIM Inversion, our methodology aimed
E.3 COMPARISON WITH MODEL-BASED EDITING
We also compare three model-basee based editing methods, InstructPix2Pix (Brooks et al. 2023), Instruct- Diffusion (Geng et al. 2023). and Blended Latent Diffusion (Avrahami et al. (2023) in Table 9 PnP Inversion added with Crompttto-Prompyy shows better structure and background preservation as well as better CLIP similarity than the two end-to-end editing models InstructPix2Pix and Instruct- Diffusion. Blended Diffusion uses an explicit mask and only performs editing in the mask. We directly use ground-truth mask in PIE-Bench as input, which leads to better background preserva- tion and CLIP similarity score. However, the forced editing makes the editing part incompatible with the background. as shown in Figure ! and thus having a much larger Structure Distance compared PnP Inversion added with Compt-to-Prompp
Metrics
Structure
Background Preservation
CLIP Similariy
Method
Distance
a 20
PSNR LPIPS vn 4 MSE 104
SSIM 10
Whole
Edit t
InstructPix2Pix
57.91
20.82
158.63
227.78
76.26
23.61
21.64
InstructDiffusion
75.44
20.28
155.66
349.66
75.53
23.26
21.34
Blended Diffusion
81.42
29.13
36.61
19.16
86.96
25.72
23.56
Ours+P2P
11.65
27.22
54.55
32,86
84.76
25.02
22.10
Table 9: Comparison of model-based editing results.
18

---

Published as a conference paper at ICLR 2024
Guidance Scale
Structure
Background Preservation
CLIP Similariy
Inverse Forward
Distance x108
PSNR LPIPS 100 J.ME
SSIM 102
Whole
Edit
0
1
6.23
28.96
40.71
22.83
86.29
23.19
20.49
2.5
8.03
28.27
44.59
25.90
85.89
23.89
21.09
0
5
10.74
27.49
50.69
30.55
85.23
24.64
21.75
0
7.5
13.38
26.86
56.66
35.29
84.56
25.04
22.08
1
4.77
29.70
36.32
19.84
86.61
22.97
20.28
2.5
5.74
29.19
39.57
21.81
86.35
23.87
21.01
5
8.76
28.04
47.59
27.45
85.52
24.65
21.76
7.5
11.65
27.22
54.54
32,86
84.76
25.02
22.10
2.5
1
20.17
25.87
67.73
49.33
83,44
21.28
19.02
2.5
2.5
8.90
28.73
43.37
25.77
85.91
23.50
20.80
2.5
5
9.01
28.27
47.05
27.77
85.55
24.62
21.70
2.5
7.5
11.36
27.35
54.43
33.40
84.76
25.09
22.15
5
1
70.65
19.85
155.46
203.41
75.92
16.38
15.84
5
2.5
45.16
22.63
111.55
123.84
79.59
19.95
18.11
5
5
28.85
25.18
79.92
70.91
82.35
22.60
20.09
7.5
23.11
25.88
71.19
54.33
83.08
24.02
21.17
7.5
1
97.45
17.62
203.49
309.26
71.92
14.38
14.79
7.5
2.5
74.55
19.43
165.31
224.15
74.84
17.19
16.34
7.5
5
56.22
21.53
131.04
154.97
77.57
19.82
18.07
7.5
7.5
47.06
22.76
113.34
120.94
79.05
21.66
19.42
Table 10: Ablation on the influence of guidance scale
E.4 INFLUENCE OF GUIDANCE SCALE
In our experiments, observed significant impact of the guidance scale the inversion and for- ward processes of DDIM, consequently affecting the editing results. Previous studies have typically employed inversion guidance scale of 1/0, coupled with empirical forward guidance scale of 7.5. However, no systematic experimental evidence determines the optimal combination of guidance scales for achieving the best editing performance, and elucidates how deviations in these guidance scales affect both reconstruction and editing. Hence, in this section, we present comprehensive analysis of systematic results addressing this matter based on Table I0 and find that, in fact, an verse guidance scale of 2.5 and a forward guidance scale of 7.5 reaches the best balance of essential content preservation and edit fidelity.
When keeping the inverse guidance scale constant, we observed that as the forward guidance scales in- creased gradually, there was an initial improvement in background preser- vation, followed by a decline. The inflection point at the inverse guidance scale being equal to the forward guidance scale (e.g., inverse with guidance scale 2.5, forward with guidance scale 2.5). In contrast, the CLIP similarity showed a consistently increasing trend.
135 1.9
11.
approximately
10.11
OA
(25.330
(15.1)
25,99
22.97
(..
(1.. 1.51
19364
(55.
CLIP CLIP Store
MSE
309.5
Figure 6 enables a clear observation of a noticeable trade-off between es- sential content preservation and edit fidelity. The abscissa represents sorted essential content preservation metrics, while the ordinate corre- sponds to the respective CLIP Sim- ilarity, aiming to illustrate the con- trasting balance between these two
36.00
100,00
PSNK
Figure 6: The impact of different inverse and forward guidance scales on evaluation metries.
19

---

Published as a conference paper at ICLR 2024
major categories of metries. Results show that inverse guidance scale of 2.5 and a forward guid- ance scale of 7.5 show the best balance of editing and preservation. The fundamental reason for this trade-off is that present editing methods lack the ability accurately differentiate between regions that require modification and those that do not. This leads to an inherent conflict where successful and precise edits lead to substantial alterations of the source image while contradictory the objec- tive of essential content preservation. This experimental observation emphasizes distinct optimal range for the guidance scale on evaluation metrics, carefully selected guidance scale can improve the alignment of the inverse and forward processes, and thus improve the editing performance.
E.5 ABLATION OF STEP AND INTERVAL
To illustrate PnP Inversion's s performance in different diffusion steps and different add back inter- vals, we further provide results of PnP Inversion added with Prompt-to-Prompl with step numbers of 20, 50, 100, and 500 in Table II] and with interval of 1, 2, 5, 10, 24, 49 in Table 12 in Table [ show that PnP Inversion is robust to different diffusion steps. Fewer steps will lead to a relatively better preservation of structure and background, bigger steps will have a better clip sim- ilarity since target text embedding brings more influence the inference process. Table 12 shows the results of performing PnP Inversion in interval steps, which leads to update delay. Results show that with the interval increase, performance would become closer to DDIM Inversion with larger structurebbackgroundd distance. When the update performed step-by-step, that is, when the interval is 1, PnP Inversion performs best in terms of the overall metrics.
Results
Metries
Structure
Background Preservation
CLIP Similarity
Steps
Distance x + PSNR LPIPS x103
MSE x104 4 SSIM x
Whole Edited t
20
10.60
27.49
51.80
30.62
84.94
24.73
21.75
50
11.65
27.22
54.55
32.86
84.76
25.02
22.10
100
12.22
27.01
56.00
34.49
84.53
25.18
22.28
500
13.00
26.82
57.44
35.76
84.39
25.30
22.42
Table 11: Ablation of different inference steps.
Metries
Structure
Background Preservation
CLIP Similarity
Interval Distance
PSNR f LPIPS x J.E
SSIM
f Whole 1 Edited f
1
11.65
27.22
54.55
32.86
84.76
25.02
22.10
2
11.83
27.15
54.53
33.32
84.67
25.06
22.11
5
13.06
26.86
57.30
35.32
84.43
25.05
22.11
10
16.18
26.03
66.37
42.24
83.60
24.98
22.13
24
24.08
24.20
77.73
66.99
82.41
24.91
22.05
49
47.05
21.21
128.64
126.34
78.19
24.81
22.22
Table 12: Ablation of performing PnP Inversion in interval steps.
E.6 RESULTS OF DIFFERENT EDITING TYPES
We provide the performance in each editing type of PnP Inversion added to Prompt-to-Promp (Hertz al., 2023) and MasaCtrl (Cao al.) 2023) Table 13 and Table [4 The results vary across different editing types and different editing methods. For both Prompt-to-Prompt and MasaCtrl, type performs quite closely line with its performance across categories, involves random volunteer-selected images and editorial instructions.
In the editing types that Prompt2Prompt struggles with, such as adding objects (type 2), deleting objects (type 3), and modifying object pose (type 5), the model shows minimal changes, resulting in relatively better evaluation result in essential content preservation metrics. Anomaly, type 5 of Commptto--Pompp shows the highest Clip Similarity lies in the insensitivity of the CLIP model on the object pose and leads to similarity in source and target prompt features. And since the source prompt is written by Blip2, which has high CLIPP
editing object poses. We infer the reason
20

---

Published as a conference paper at ICLR 2024
similarity to the source image, the images with minor alterations in type 5 tend to have better CLIP Similarity to the source prompt. Type 8 (change background) and (change style) have a bad Structure Distance because the areas that need modification relatively large. For type 9, the whole image is required for editing. Thus, we do not report background preservation metric and Whole Image CLIP Similarity, which is the same Edit Region Clip Similarity.
MasaCtrl demonstrates proficiency modifying object poses, leading to relatively better results in the respective category 5. Moreover, we find that MasaCtri also good adding and deleting objects, which requires significant image modifications and structure changes. We attribute this reason to the Mutual Self-Attention mechanism of MasaCtrl, which facilitates the model's ability to extract texture information from the source branch and structural information from the target branch. Therefore, more friendly to modifications sensitive structural information changes.
Metrics
Structure
Background Preservation
CLIP Similarity
Type
Distance 2103 + PSNR LPIPS xio2
MSE 1io J SSTM x102
Whole Edited f
0
11.73
28.58
50.07
30.18
85.87
25.09
22.66
I
12.60
26.74
55.76
29.75
84.34
24.48
20.11
3 10.09
27.55
53.68
30.71
86.72
25.12
23.20
23.97
78.78
53.06
78.37
23.76
17.56
4
12.59
28.12
51.97
23.13
85.85
24.85
22.58
5 10.08
26.57
60.23
33.42
82.98
26.24
22.38
6 10.53
26.40
55.34
35.91
83.54
25.55
20.95
11.21
30.39
40.38
17.47
89.14
25.81
8 12.23
27.40
39.03
30.70
87.68
24.34
22.01
9 14.50
26.04
Table 13: Performance of PnP Inversion added with Prompt-to-Promp on editing types.
Metries
Structure
Background Preservation
CLIP Similarity
Type
textbfDistance,, x103
PSNR
LPIPS 100
MSE x 104
SSIM 1002
1 Whole
Edited f
23.48
23.76
81.72
75.25
82.67
23.66
21.27
33.26
21.03
106.87
103.16
79.25
24.27
19.59
23.26
78.73
67.14
83.61
25.04
22.73
21.55
22.44
92.66
89.60
79.16
26.38
22.31
23,60
23.51
94.72
66.43
82.08
24.52
21.55
26.53
23.79
70.81
59.32
85.89
25.44
23.44
25.79
21.65
97.36
98.51
78.64
24.62
19.61
23.63
23.49
62.20
61.98
85.27
23.39
21.55
26.87
21.05
104.79
97.34
76.11
24.16
17.67
22.53
24.49
Table 14: Performance of PnP Inversion added with MasaCtrl on editing types.
F QUALITATIVE RESULTS
Due to the page limit, we do not provide lots of visualization results in the main paper. In this section, we provide a comparison of visualization for further verification of quantitative results.
Comparison with different inversion- editing. Figure 8 shows the comparison of different inversion methods combined with Prompt-to to-Prompt (Hertz et al.) 2023). Figure 9 Figure I0 and Figure shows the visualization results of MasaCtrl (Cao et al. (023). Pix2Pix-Zen (Parmar al. 2023 and Plug-and-Plaa (Tumanyan et al. 2023) w/ and w/o PnP Inversion.
Comparison with essential content preservation methods. Visualization results of essential content preservation methods are shown in Figure 12 including Proximal Guidance (Han et al.
2023 Edit Friendly DDPM (Huberman ShubermannSSiegelllaae 2023), and EDICT (Wallace al. 2023)
21

---

Published as a conference paper at ICLR 2024
Comparison with model-based editing. Visualization results of model-based editing are shown in Figure 13 including InstructPix2Pix (Brooks et al. 2023), InstructDiffusion (Geng et al. 2023),
and Blended Latent Diffusion (Avrahami et al. 2023).
G LIMITATIONS AND FUTURE WORKS
Since the performance of PnP Inversion is strongly connected to existing diffusion-based editing methods, our method inherits most of their limitations. Although PnP Inversion boosts existing editing techniques' performance, it is still unable bring about fundamental changes in the editing model performance, which is unstable, with a low success rate. In Figure we have chosen specific cases in which Blended Latent Diffusion, along with the ground truth mask, succeeds, whereas all other editing methods fail. This demonstrates the inherent capability of diffusion models to per- form corresponding edits. However, existing diffusion -aseed editing algorithms lack the capability of realization without giving explicit masks. Moreover, although PnP Inversion leads to better performance on average, success not guaranteed every case.
A woman in shorts + A woman in shorts looking at right side
A logo of bird shape A logo of x shape
(a) Source Image Blended Diffus OurstMasa
(b)
(c)
(d) Ours-P2P
(c) Ours Zero
(1) Ours PnP
Figure 7: Visualization results of failure cases in existing certusion-based editing methods. (a) source image; (b) Blended Latent Diffusion (Blended Diffus) (Avrahami et al. (023); (c) PnP In- version added to MasaCtrl (Masa) (Cao et al. 2023); (d) PnP Inversion added to Prompt-to-Prontt (P2P) (Hertz al., 2023); (e) PnP Inversion added to Pix2Pix-Zero (Zero) (Parmar et al.) 2023);
(f)PnP Inversion added to Plug-and-Plas (PnP) (Tumanyan et al. 2023). The source and target prompt are shown the top of each row.
PnP Inversion may also lead ethical issues that are worthy of consideration. The data used in the training of diffusion models unavoidably contain personally identifiable information, social biases, and violent content, which will also influence the editing results of our model. PnP Inversion be misused or modified to produce contradictory results and lead to potential negative societal impacts (e.g., arbitrary modification on private photo). We believe these issues should considered, and need to design and engineer AI capabilities to fulfill their intended functions while possessing the ability detect and avoid unintended consequences and unintended behavior.
We hope that this work can motivate future research with a focus on diffusion-based editing for higher essential content preservation and edit fidelity. Specifically, future directions include but are not limited to (1) an extension to video editing, (2) editing models with higher success rates and more editable scenes, and (3) more comprehensive metric evaluation system evaluate the effectiveness of editing.
22

---

Published as a conference paper at ICLR 2024
cat sitting next to mérror A silver eat sculpture silting next to mirror
An orange cat sitting top of fonce A Black cat sitting top of fence
A dog stands with mouth closed A dog stands with mouth opened
Two birds silling on branch Two origami biruh silling on branch
sitting with beads collar sitting with beads collar putting hands down
A meerkat puppy wrapped in a blue towel A ßon puppy wrapped in a blue towel
A woman jacket standing the rabn A women blouse standing in the rain
A puinting of a woman holiding pink flower A painting of woman holding teddy bear
woman gold makrup
woman makeup
(a) Image DDIM
(b)
(c)
(d)
(e)
(i)
StyleD
Figure 8: Visualization results of different inversion methods combined with Prompt-to- Prompt (P2P) (Hertz et al., 2023). The source image is shown in col (a). We compare (f) PnP Inversion with different inversion techniques: (b) DDIM Inversion (DDIM) (Song al.) 2020), (c) Null-Text Inversion (NT) (Mokady et al. 2023), (d) Negative-Prompl Inversion (NP) (Miyake et al., 2023), and (e) StyleDiffusion (StyleD) (Li al. (002b)). The source and target prompt are shown at the top of each row. The improvements are mostly tangible, and we circle some of the subtle discrepancies w/o PnP Inversion red.
23

---

Published as a conference paper at ICLR 2024
Painting of a shepherd dog
vnd A park bench and rod trees
Painting a poodle dog
A. park bench and pink trees
A mouse and a leaf on the ground mouse on ground
A woman sitting at table with bottle of water woman ritting at table
A kitten playing with balls A kitten
An older man sitting with dog An older man sitting
cat cat with hat
A sitting at table with laptop
Ron sitting table
man with fils hands hanging down A man with his hands rasing
A cat in low polygonal style A fox in a low polygomal style
A golden retriever holding flower A goldan retriever
A dog laying on a white background fion laying on while background
An old man playing an electric gultar An old man playing an electric plano
Cartoon, dag laying down on the ground
Cartoon dog jumping up from the ground
A painting of a rat with eed patnting pig with eyes
Vase with sunflowers pears on pears
Vase with sunflowers and bananas table
(a)
(b)
(c) Masa
(d)
(e)
(
Souree Image DDIM
Source Image DDIM
Masa
Figure 9: Visualization results of MasaCti (Masa) (Cao et al. 2023 w/ and w/o PnP Inversion. The source image shown in col (a) and (d). The col (b) and (e) show results w/o PnP Inversion. The col (c) and (f) show results w/ PnP Inversion. The source and target prompt shown the top of each row.
24

---

Published as a conference paper at ICLR 2024
digital woman with curly hair
serious man
A digital art woman with straight hair
A angry
vound paintiig of forest round painting forest with deer
famitty playing A family playing with clouds in the sky
Plant and notebook white Plant and calendar white tabíe
A tiger swimming a poná of green algae
dog swinming in pond of green algae
painting of rat with red A painting of a plg with eyes
Yasse with sunflowers and pears on table sunflowers bananas on
the roof night Dog the reef nighit
monkey wearing goggles man wearing colorful goggles
AT
A cup with smoke coming out of it cup a flower coming
Cat with yellow eyes Cat green eyes
Mother duckk and babies Sketch painting of mother duck and babies
A woman in a jacket stansfing in the rain A woman a blouse standinn in the rain
playing with flower dog is playing with flower
WWidte tiger brown ground White cat on brown, ground
(d)
Source Image DDIM + Zero Ours Zero
Source Image DDIM Zero Ours + Zero
Figure 10: Visualization results of Pix2Pix-Zero (Zero) Parmar et al. 2023) w/ and w/o PnP Inversion. The source image is shown in col (a) and (d). The col (b) and (e) show results w/o PnP Inversion. The col (c) and (f) show results w/ PnP Inversion. The source and target prompt are shown the top of each row.
25

---

Published as a conference paper at ICLR 2024
Sea and house
A Birdd standing on clods bird standing on eggs
Forest
A sitting table window
woman ine fiving Watercolor of a woman in living room
A. plant sitting on a table near a window
woman with dark hatr woman dark hair wearing earrings
A woman with garland on head woman with hat
white bunmy cup A watercolor of pink Funny in cup
white with wand and
A purple mouse with wand and star
A young man with flue eyes young with blue eyes
robin on a branch sfiver robin sculpture branch
white Cartoon woman in white
chalr with table A chair with table in blue background
"While horse running in field Watercolor of wfitte horse raming in fieli
A waman in blue dress Piscel art woman in Blue dress
An orange cat sitting on top of a fence Black sitting on top fence
dog laying on white background flon laying on whitte background
(a) Image DDIM
(b)
(c)
(d) Image
(e)
(O
PaP
DDIM
Figure 11: Visualization results of Plug-and-Play (PnP) Tumanyan et al. 2023 w/ and w/o PuP Inversion. The source image shown in col (a) and (d). The col (b) and (e) show results w/o PnP Inversion. The col (c) and (f) show results w/ PnP Inversion. The source and target prompt shown at the top of each row.
26

---

Published as a conference paper at ICLR 2024
Woman walking down cobélestone street Woman walking down a thick forest
Beautiful woman with clean background
Beautiful woman with blue background
Picnic table
"aatrcooorr of pionic table
The galaxy over the Durdle door - The sunset over the Durdle door
Woman with brown hair
Woman with blue hair
AAAA
A woman with castle in the background A woman with ghost in the Background
A rose the rail A rose on the wood
Red berriiss tree branch
Purple Berrries tree branch
(b) NT Prox
(c) NP+ Prox
(d) EF + P2P
(c)
(f) Ours + P2P
Source Image
EDICT P2P
Figure 12: Visualization results of different essential content preservation methods. The source
image is shown in col (a). We compare (f) PnP Inversion added to Prompt-to-Promp (P2P) (Hertz
al. 2023) with different combined methods: (b) Null-Text Inversion (NT) (Mokady et al. 2023)
added to Proximal Guidance (Prox) (Han et al., 2023). (c) Negative-Prompt Inversion (NP) (Miyake)
al. 2023) added Proximal Guidance, (d) Edit Friendly DDPM (EF) Clluberman-Spiegelle
et al. 2023) added to Soompt-to-Prompp and (c) EDICT (Wallace et al. (023) added to Prompt-to-
Prompt. The source and target prompt are shown the top of each row.
27

---

Published as a conference paper at ICLR 2024
notebook and camera map A notebook and camera carpet
Cat
roof at night
Dog on the roof night
O a
A girl and her dog in the field girl and her monkey in the field
A young woman holding dog A old woman is holding dog
A girl sitting
table with pizza A girl sitting table with noodles
woman
A riding hon
White tiger on brown ground White cat on brown ground
A cup of coffee with tully drawing cup of coffee with fon drawing
(4)
bb) InstructP2P
(d)
(c)
Source Image
InstructDiffus
Blended Diffus Ours
Figure 13: Visualization results of different essential content preservation methods. The source image is shown in col (a). We compare (f) PnP Inversion added to Prompt-to-Prompt (P2P) (Hertz) al. 2023) with different model-based editing methods: (b) InstructPix2Pix (InstructP2P) (Brooks al. 2023), (c) InstructDiffusion (InstructDiffus) (Geng et al. 2023) and (d) Blended Latent Dif- fusion (Blended Diffus) (Avrahami et al. 2023). The source and target prompt are shown the top of each row.
28


Reviews:
[{"review_id": "hoZCeazwM7", "reviewer": "Reviewer_2rho", "summary": "The paper presents \"direct inversion\", a general inversion technique to improve essential content preservation and edit fidelity of diffusion-based image editing methods. An editing benchmark is also proposed for performance evaluation.", "questions": "please see my questions in weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and well-presented with nice figures.\n- The results are pleasing to look at and are convincing.\n- The proposed method is simple and effective.\n- Dataset/evaluation benchmark contribution.\n- The experiments are comprehensive. The proposed method is quite general and is evaluated on 8 recent editing methods.", "weaknesses": "- The method section 4.2 is not very clear to me, especially the bracket notations in the algorithm box. It would be helpful to explain lines 3, 7-9 in more detail.\n- It might be worth adding discussion and comparison of a related but concurrent work [1].\n- The name \"direct inversion\" clashes with another existing work [2], which might cause ambiguous.\n- Typo: Algorithm 1, Part I: \"Invert\" z_0^{src}; sec 4.2, \"optimization-based\" inversion.\n- The paper shows promising empirical results but is still not theoretically motivated.\n\n[1] Pan, Zhihong, et al. \"Effective Real Image Editing with Accelerated Iterative Diffusion Inversion.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[2] Elarabawy, Adham, Harish Kamath, and Samuel Denton. \"Direct inversion: Optimization-free text-driven real image editing with diffusion models.\" arXiv preprint arXiv:2211.07825 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0PPlNbcmT", "reviewer": "Reviewer_WzRy", "summary": "This paper proposed a method for Diffusion based inversion and editing, where all the intermediate generated zt values are stored and then utilized for sampling new samples based solely on the difference from the zt values generated during the sampling process. This approach is easy to apply to all editing methods. The method demonstrated its applicability to P2P, MasaCtrl, P2P-Zero, and PnP by calculating the difference between the zt values during the editing process (with those methods) and the zt values obtained from the original DDIM inversion. To evaluate this method, the paper introduced PIE-Bench, which used 700 images divided into 10 categories for editing to showcase the preservation of structure, background, and CLIP similarity.", "questions": "Please see the weakness part.\n\nEspecially I'm wondering about the Algorithm.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "This paper provided well-organized evaluation criteria, encompassing 10 different categories that include tasks such as changing or adding and removing objects, altering poses, changing colors, modifying materials, and changing backgrounds. Additionally, editing masks are also provided.\n\nBased on these evaluation criteria, the paper presented a wide array of numerical evaluation results, proving superior performance across all categories. A detailed ablation study was provided, and in the supplementary material, various experimental setups and their results were thoroughly documented, offering valuable insights and enhancing reproducibility.", "weaknesses": "First and foremost, I would like to make a strong suggestion to the authors. The content of Figure 3 seems to be largely irrelevant to the content of the paper. While I express my utmost gratitude for the detailed explanation and organization of previous works, Figure 3 does not play a significant role in aiding understanding. Instead, I would prefer if Figure 5 from the supplementary material were included in the main text. Additionally, a more detailed explanation of the benchmarks and evaluation metrics in the main body of the text would be beneficial. This information is considered one of the major contributions of the paper, yet it is not present in the main text.\n\nSecondly, the explanation of the method is unclear. There are no definitions provided for what the brackets [ ] mean in lines 3, 7, 8, and 9 of Algorithm 1, or what o_t represents. There is also a need for an explanation on whether z_t encompasses both src and tgt. If my understanding based on the code is correct, this paper stores all the zt values generated during DDIM inversion, uses them to calculate a small editing direction at each step, and then reflects this in the z^tgt used to generate the actual results. This algorithm feels somewhat similar to the approach used in CycleDiffusion [https://arxiv.org/abs/2210.05559]. A clearer explanation of the algorithm would greatly assist in correcting my understanding.\n\nThirdly, the benchmark is divided into 10 categories, but scores for each category are not reported. I am particularly interested in the scores for the category involving pose changes. I suspect that most of the proposed methodologies would struggle with changing poses. A discussion and reporting of scores on this matter would be appreciated, at least in the supplementary material.\n\nMinor point: Regarding Figure 4, it is disappointing that the only result shown with our method applied is Ours+P2P. (But I saw additional results in Supple.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ERDcBtV7Hj", "reviewer": "Reviewer_C1kK", "summary": "This paper proposes a simple but effective method that improves existing diffusion-based image editing methods. The method is very easy to implement, with only three lines of code. However, it helps resolve the discrepancy between source latent and target editing latent for many diffusion-based editing methods. This helps preserve essential content and maintain editing flexibility. The paper also presents a comprehensive image editing benchmark PIE-Bench covering ten editing types.", "questions": "1. What is the meaning of step 1, 2 in Figure 2? \n2. In the algorithm1, what is the meaning of argument $[C^{src}, C^{tgt}]$ in the DDIM_Forward function call? Is it suggesting that the forward function is called twice, with one of the calls on $C^{src}$ and the other on $C^{tgt}$? If so, why line 9 function call has only one output?\n3. I am confused about how the source branch interacts with the target branch. It seems that the $z_{t}^{tgt}$ is only updated using $z_{t+1}^{tgt}, C^{tgt}$ without any source branch information. Could authors clarify line 9 in the algorithm? Is it related to the implementation of $DDIMForward_{Editing_Model}$?\n4. What's the number of images for each editing type in the PIE-Bench creation? As far as I understand, most of the editing types are local region editing, e.g., change object, add object, and large region editing is limited to style change only. I wonder if the dataset is mostly local editing images.\n5. What is the input text for CLIP similarity evaluation in **Whole** and **Edit**? As far as I understand, the target prompt in the PIE-Bench is a full description of the target image instead of the specification of the local region.\n6. The CLIP Similarity for different methods seems very close in Table 1, 4, and 8. Is it possible the CLIP similarity cannot distinguish the editing quality? Could authors include BLIP similarity or human evaluation to make evaluation more comprehensive?\n\nI would increase my rating if the authors could resolve the questions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper presents a thorough investigation of existing diffusion-based prompt editing methods and identifies that previous methods improve essential content preservation through fine-tuning a learned variable to rectify the distance between the source branch latent and the target branch latent and propose a simple rectification method.\n2. The proposed rectification method is simple but effective and is suitable for a large amount of diffusion-based editing methods. This paper presents comprehensive experiments to apply their method to different methods and see a universal improvement in both essential content preservation and edit fidelity.\n3. This paper presents a comprehensive diffusion-based editing benchmark covering ten editing types and 700 human-reviewed high-quality samples with source prompt, target prompt, source image, and editing region annotation, which is helpful for future study.", "weaknesses": "1. The writing in the method part is hard to follow. I would suggest authors use meaningful subscripts to denote source latent, source prompt forward latent, and other patents instead of $z_0, z_t', z_{t}''$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NQMzAiT63B", "reviewer": "Reviewer_jyta", "summary": "The $\\textbf{Direct Inversion}$ technique separates the source and target diffusion branches, enhancing content preservation and edit fidelity. It surpasses previous methods and substantially accelerates editing, as evidenced by the PIE-Bench benchmark.", "questions": "1. In the column PnP of Fig.1, PnP doesn't do a good job in preserving content, sometimes texture and shape is hallucinated. Using direct inversion can correct them. But why not background color in the third row? \n\n2. Following Fig.2, \"...This results in a learned latent with a discernible gap between $z_0^{''}$ and the original $z_0$...\". So how does the optimized $z_0^{''}$ deviates from original distribution? It's not quite clear how deviation happens, what does it look like, and why it negatively affects performances. Could authors provide concrete examples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a novel technique called Direct Inversion, which tackles the problem of balancing content preservation and edit fidelity in previous works. Motivation is clear and presentation is in a roughly good shape overall.\n\n1. Direct Inversion is neat and can be used as a plug-and-play into popular optimizaiton-based diffusion editing methods to enhance the performances.\n\n2. The paper provides a comprehensive and well-structured review of existing literature. Analysis of each method makes the motivation strong and presentation clear.\n\n3. Experimental results are sound and analysis is rigorous.\n\n4. Authors also provide a editing benchmark, called PIE-Bench, which is believed to benefit future works.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)