Object-Aware Inversion and Reassembly for Image Editing
Authors: Zhen Yang
Ganggui Ding
Wen Wang
Hao Chen
Bohan Zhuang
Chunhua Shen
Venue: ICLR 2024 poster (2024)
Keywords: Image editing
Diffusion model
Primary area: generative models

Abstract: Diffusion-based image editing methods have achieved remarkable advances in text-driven image editing. The editing task aims to convert an input image with the original text prompt into the desired image that is well-aligned with the target text prompt. By comparing the original and target prompts, we can obtain numerous editing pairs, each comprising an object and its corresponding editing target. To allow editability while maintaining fidelity to the input image, existing editing methods typically involve a fixed number of inversion steps that project the whole input image to its noisier latent representation, followed by a denoising process guided by the target prompt. However, we find that the optimal number of inversion steps for achieving ideal editing results varies significantly among different editing pairs, owing to varying editing difficulties. Therefore, the current literature, which relies on a fixed number of inversion steps, produces sub-optimal generation quality, especially when handling multiple editing pairs in a natural image.
To this end, we propose a new image editing paradigm, dubbed Object-aware Inversion and Reassembly (OIR), to enable object-level fine-grained editing. Specifically, we design a new search metric, which determines the optimal inversion steps for each editing pair, by jointly considering the editability of the target and the fidelity of the non-editing region. We use our search metric to find the optimal inversion step for each editing pair when editing an image. We then edit these editing pairs separately to avoid \concept. Subsequently, we propose an additional reassembly step to seamlessly integrate the respective editing results and the non-editing region to obtain the final edited image. To systematically evaluate the effectiveness of our method, we collect two datasets called OIRBench for benchmarking single- and multi-object editing, respectively. Experiments demonstrate that our method achieves superior performance in editing object shapes, colors, materials, categories, \textit{etc.}, especially in multi-object editing scenarios.
The project page can be found in https://aim-uofa.github.io/OIR-Diffusion/.

Published as a conference paper at ICLR 2024
OBIECT-AWARE INVERSION AND REASSEMBLY FOR IMAGE EDITING
Zhen Yang' Ganggui Ding'
Wen Wang'
Hao Chen't If
Bohan Zhuang2t
Chunhua Shen
1 Zhejiang University, China
{zheny. ca, dingangui, wwenxyz, haochen. cad, chunhuashen) edu. on
2 Monash University, Australia
bohan. zhuang@monash.
ABSTRACT
Diffusion- image editing methods have achieved remarkable advances in text-driven image editing. The editing task aims to convert an input image with the original text prompt into the desired image that well-aligned with the tar- get text prompt. By comparing the original and target prompts, we can obtain numerous editing pairs, each comprising an object and its corresponding editing target. To allow editability while maintaining fidelity to the imput image, existing editing methods typically involve a fixed number of inversion steps that project the whole input image to its noisier latent representation, followed by denoising process guided by the target prompt. However, we find that the optimal number inversion steps for achieving ideal editing results varies significantly among dif- ferent editing pairs, owing to varying editing difficulties. Therefore, the current literature, which relies on fixed number of inversion steps, produces sub-optimal generation quality, especially when handling multiple editing pairs in a natural image. To this end, we propose new image editing paradigm, dubbed Object- aware Inversion and Reassembly (OIR), to enable object-level fine-grained edit- ing. Specifically, we design new search metric, which determines the optimal inversion steps for each editing pair, by jointly considering the editability of the target and the fidelity of the non-editing region. We use our search metric to find the optimal inversion step for each editing pair when editing image. We then edit these editing pairs separately to avoid concept mismatch. Subsequently, we propose additional reassembly step seamlessly integrate the respective edit- ing results and the non-editing region to obtain the final edited image. To system- atically evaluate the effectiveness of our method, collect datasets called OIRBench for benchmarking single- and multi-object editing, respectively. Ex- periments demonstrate that our method achieves superior performance in editing object shapes, colors, materials, categories, etc., especially in multi-object editing scenarios. The project page can be found here.
INTRODUCTION
Large-scale text-to-image diffusion models, such as Latent Diffusion Models (Rombach et al., 2022). SDXL (Podell et al., 2023), Imagen (Saharia et al., 2022), DALL-E 2 (Ramesh et al., 2022), have advanced significantly and garnered widespread attention. Recently, many methods have be- gun using diffusion models for image editing. These methods offer tine-grainer control over content, yielding impressive results that enhance the field of artistic content manipulation. We focus on text- driven image editing, aiming to align the region of interest (editing region) with user-defined text prompts while protecting the non-editing region. We define the combination of the editing region and its corresponding editing target the "editing pair". In Fig. (parrot, crochet parrot) emerges as an editing pair when comparing the original prompt with target prompt 1. To enable editability in the editing region while maintaining fidelity to the input image, existing text-driven image editing methods (Tumanyan et al., 2023; Couairon et al., 2022; Hertz et al., 2022; Mokady et al., 2023;
HH and BZ. corresponding authors.

---

Published as a conference paper at ICLR 2024
Optinal Resill
0 blue and yellow cnochet porret sitting branch
Original Image
Editing Pair:
parrot)
Civersion Stopo2t
Optinal Ricid
Target Peamet 2:
attting
aa
gold bronch
Editing Pein:
branch. gold branch)
Invrssoon Stept
Target Prempt 3:
bbblx yellow crechet parrot
parret sitting on branch
Original Prompti yellow
Editing Peirs:
atting
[aarel, crochet parrot)
(o bronch, branch)
Concept Bditinn
30 35
40
45
Inversion
Step
Figure Motivation. In process of text-driven image editing, we first inverse the original image progressively acquire all latents. Then, we denoise each latent to generate images under the guidance of the target prompt. After obtaining all the images, the most optimally edited results selected by human. From the first and second rows, note that different editing pairs have unique optimal inversion steps. Moreover, we observe editing different editing pairs with the same inversion step results in concept mismatch poor editing, shown in the third row.
Meng et al., 2021) typically project the original image into its noisier representation, followed by denoising process guided by the target prompt.
Our key finding is that different editing pairs require varying inversion steps, depending the edit- shown the first and second rows in Fig. the object and target within an editing pair are similar, it requires only few inversion steps, and vice versa. Over-applying inver- sion steps to easy editing pairs or insufficient steps to challenging pairs can lead a deterioration in editing quality. This be even ing these objects with the same inversion step at once can lead to concept mismatch or poor editing in the third row of Fig. 1. However, current methods (Tumanyan al., 2023; Couairon al., 2022; Hertz et al., 2022: Mokady et al., 2023; Wang et al., 2023) uniformly apply fixed inversion step to different editing pairs, ignoring the editing difficulty, which results in suboptimal editing quality.
ing difficulties.
when multiple editing pairs exist in the prompt, edit-
To this end, we propose a novel method called Object-aware Inversion and Reassembly (OIR) for generating high-quality image editing results. Firstly, design a search metric Fig. 2. This met- ric automatically determines the optimal inversion step for each editing pair which jointly considers the editability of the editing object interest and the fidelity the original image of the non-editing region. Secondly, shown in Fig. 3, we propose a disassembly then reassembly strategy to enable generic editing involving multiple editing pairs within an image. Specifically, we first search the optimal inversion step for each editing pair with our search metric and edit them separately, which effectively circumvents concept mismatch and poor editing. Afterward, we propose an additional reassembly step during denoising to scamlessly integrate the respective editing results. In this step, simple yet effective re-inversion process introduced to enhance the global interactions among edit- ing regions and the non-editing region, which smooths the edges of regions and boosts the realism of the editing results.
To systematically evaluate the proposed method, collect two datasets containing 208 and 100 single- and multi-object text-image pairs, respectively. Both quantitative and qualitative exper- iments demonstrate that method achieves competitive performance in single-object editing, and outperforms state-of-the- (SOTA) methods by large margin in multi-object editing scenarios.
In summary, key contributions
follows.
We introduce a simple yet effective search metric to automatically determine the optimal inversion step for each editing pair, which jointly considers the editability of the editing object of interest and the fidelity to the original image the non-editing region. The process of using a search metric select the optimal result can be considered new paradigm for image editing.
2

---

Published as a conference paper at ICLR 2024
We design novel image editing paradigm, dubbed Object-aware Inversion and Reassembly, which separately inverses different editing pairs to avoid concept mismatch or poor editing and subsequently reassembles their denoised latent representations with that of the non-editing region while taking into account the interactions among them.
We collect two new image editing datasets called OIRBench, which consist of hundreds of text- image pairs. Our method yields remarkable results, outperforming existing methods in multi- object image editing and being competitive to single-object image editing, in both quantitative and qualitative standings.
RELATED WORK
Text-driven image generation and editing. Early methods for text-to-image synthesis (Zhang et al., 2017; 2018b; Xu et al., 2018) are only capable of generating images in low-resolution and limited domains. Recently, with the scale-up of data volume, model capacity, and computational resources, significant progress has been made in the field of text-to-image synthesis. Representative methods like DALLE series (Ramesh et al., 2021: 2022), Imagen (Saharia et al., 2022), Stable Diffusion (Rombach et al., 2022), Parti (Yu et al., 2022). and GigaGAN (Kang et al., 2023) achieve unprecedented image generation quality and diversity in open-world scenarios. However, these methods provide limited control over the generated images. Image editing provides finer-grained control over the content of an image, by modifying the user-specified content in the desired manner while leaving other content intact. It encompssse many different tasks, including image colorization (Zhang et al., 2016), style transfer (Jing et al., 2019), image-to-image translation (Zhu et al., 2017), etc. We focus on text-driven image editing, as it provides a simple and intuitive interface for users. We refer readers to (Zhan et al., 2023) for comprchensive survey multimodal image synthesis.
Text-driven image editing. Text-driven image editing need understand the semantics in texts. The CLIP models (Radford et al., 2021), contrastively pre- pre-trained with learning on internet-scale image- text pair data, provide a semantic-rich and aligned representation space for image and text. There- fore, several works (Abdal et al., 2020; Alaluf et al., 2021; Bau al., 2020; Patashnik et al., 2021) attempt to combine Generative Adversarial Networks (GANs) (Goodfellow et al., 2020) with CLIP for text-driven image editing. For example, StyleCLIP (Patashnik et al., 2021) develops text in- terface for StyleGAN (Karras et al., 2019) based image manipulation. However, GANs are often limited in their inversion capabilities (Xia et al., 2022), resulting in an undesired change in image.
The recent success of diffusion models in text-to-image generation has sparked a surge of interest in text-driven image editing using diffusion models (Meng et al., 2021; Hertz et al., 2022; Mokady et al., 2023; Miyake et al., 2023; Tumanyan et al., 2023; Avrahami et al., 2022; Wang et al., 2023; Brooks et al., 2023; Kawar et al., 2023). These methods typically transform an image into noise through noise addition (Meng et al., 2021) or inversion (Song et al., 2020), and then performing de- noising under the guidance of the target prompt to achieve desired image editing. Early works like SDEdit (Meng et al., 2021) achieve editability by adding moderate noise to trade-off realism and faithfulness. Different from SDEdit which focuses on global editing, Blended Diffuusion (Avrahami al., 2022) and Blended Latent Diffusion (Avrahami al., 2023) necessitates local editing by using a mask during the editing process and restricting edits solely to the masked area. Similarly, DiffEdit can automatically produce masks and considers the degree of inversion as a hyperparameter, fo- cusing solely on the editing region. Prompi2Prompt (Hertz et al., 2022) and Plug-and-Play (PNP) (Tumanyan al., 2023) explore attention/feature injection for better image editing performance. Compared to Prompt2Prompt, PNP can directly edit natural images. Another line of work explores better image reconstruction in inversion for improved image editing. For example, Null-text Inver- sion (Mokady et al., 2023) trains a null-text embedding that allows more precise recovery of the original image from the inverted noise. Negative Prompt Inversion (Miyake et al., 2023) replaces the negative prompt with the original prompt, thus avoiding the need for training in Null-text Inversion.
While progress has been made, existing methods leverage a fixed number of inversion steps for image editing, limiting their ability to achieve optimal results. Orthogonal to existing methods, we find that superior image editing can be achieved by simply searching the optimal inversion steps for editing, without any additional training or attention/feature injection. Our approach is completely training- free and automatically searches the optimal inversion steps for various editing pairs within an image, enabling fine-grained object-aware control.
3

---

Published as a conference paper at ICLR 2024
Condidate Images
Inversion Trajectory
Paloriginal promptt: sitting the desk PyCtorget prompt); Tiger is sitting on the desk
Editing Pair: (cat, tiger)
Denoise Trajectory
Optirnal Inwersce Shep
Original Image
P(mnage)
P(noise)
Nagettive
Mac
Sue
Condidate Images
Begiun Musk
9
Outiedd Besan
Originl Treage
Wesk Gonerottr
Region Mork
P.
M,
CLIP Score Aix Max Marelizoooo
It
Inversion Step
P
(o) Candidate Images Generction
(b) Optimal Condidate Selection
Figure 2: Overview of the optimal inversion step search pipeline. (a) For an editing pair, we obtain the candidate images by denoising each inverted latent. (b) We use mask generator to jointly compute the metrics S. and Sne, and finally we obtain S by computing their average.
3 METHOD
In general, an image editing task can expressed as a triplet (Lo, Po, Pt), where Po the original prompt describing the original image I., and P. is the target prompt reflecting the editing objective. In image editing, we aim to edit Io to the target image I, that aligns with P.. To achieve this, we employ Stable Diffusion (Rombach et al., 2022), strong text-to-image diffusion model, to enable text-driven image editing. Specifically, To is first inverted to moise using DDIM Inversion guided by Po. Following that, noise is denoised generate I, guided by P, meet the user's requirement. We further define an editing pair as (O., Ot), where O2 and O, denote an object in P. and its corresponding editing target P.. respectively. As shown in Fig. there exist multiple editing pairs {OO.. }.)} given an image editing task (1 P., Pt) that have multiple editing targets.
As shown in Fig. 1, each editing pair can have distinct optimal inversion step. Hence, using single inversion step for an image with multiple editing pairs might lead to poor editing and concept mismatch. For example, the gold branch confusingly replaced with a crochet branch the 40th step in the third row of Fig. 1. In Sec. 3.1, We propose optimal inversion step search method to automatically search for the optimal inversion step for each editing pair, and in Sec. 3.2, we propose Object-aware Inversion and Reassembly (OIR) to solve the problems of poor editing and concept mismatch.
3.1 OPTIMAL INVERSION STEP SEARCH
Candidate images generation. DDIM Inversion sequentially transforms an image into its corre- sponding noisier latent representation. The diffusion model can construct an edited image I from each intermediate result from inversion step as shown in Fig. 2 (a). This process produces a set of images {IH} called candidate images. Notably, from these candidate images, one can manually se- lect visually appealing result I;" that aligns closely with P.. with its non-editing region unchanged. Its associated inversion step i* is defined as the optimal inversion step. Surprisingly, comparing to commonly used feature-injection based image editing methods (Tumanyan et al., 2023), sim- ply choosing good 1 often produces better result. A discussion on the difference between method and feature-injection--all based image editing methods can be found in Appendix A.3.
Optimal candidate selection. Since manually choosing I can be impractical, further devise searching algorithm as shown in Fig. 2 (b). To automate the selection process, we first apply mask generator to extract the editing region mask M, and the non-editing region mask Mne from Io. By default, we employ Grounded SAM (Liu et al., 2023; Kirillov et al., 2023) for mask generation. follow However, other alternatives can be used to obtain the editing mask, for example,

---

Published as a conference paper at ICLR 2024
DiffEdit (Couairon et al., 2022) MasaCtrl (Cao al., 2023) to generate masks from the attention maps. For a detailed discussion of the mask generation process, please refer to Appendix A.5.
Subsequently, propose a quality evaluation metric based on two criteria: Se, the alignment be- tween the editing region of the target image 1, and the target prompt Pt; and Sne, the degree of preservation of the non-editing region relative the original image To-
For the first criterion regarding the editing region, we utilize CLIP score (Hessel et al., 2021) to assess alignment:
CLIP image (4,, Me) CLIP te (P.)
Se Pt. Mc) = normalize(
(1)
[CLIP smage (4,, Me)ll ||CLIP teze (P))l2
where CLIP smage (It, Me) and CLIP tent (Pt) are the extracted editing region image feature and text feature with CLIP (Radford et al., 2021). normalize(-) is the min- max normalization. The normalization formula is given by: ({Sc); b min m (S.)))(mar((....
(S:)). {Sa) denotes the complete set of Se values obtained from all candidate images, i denotes the index of the image in candidate images. Insufficient inversion can restrict the editing freedom while too much inversion can lead corrupted results. Thus, we observe that S. first rises then drops as the inversion step increases.
To measure the similarity between the non-editing regions of 1 and lo we employ the negative mean squared error:
Sne (1,, Io, Mac) normalize
Io) 1n.
where denotes the element-wiss product, Mne represents the non-editing region mask. Sue usu- ally decreases as inversion step grows, since the inversion process increases the reconstruction diffi- culty of the non-editing region. The search metric simply an average of Se and Sne:
= 0.5 (Se + Sne),
(3)
where S is the search metric. As shown in Fig. 2 (b), we define the inversion step that has the highest search metric as the optimal inversion step.
Acceleration for generating candidate images. We notice that the sequential steps in generating multiple candidate images Fig. 2 (a) are independent, and the varying number steps make parallelization challenging. Consequently, we propose splicing strategy over the denoising process. Firstly, we pair denoising processes of different steps to achieve equal lengths. In this way, denoising processes of the same length can proceed simultaneously for parallel acceleration, as there is no dependency between denoising processes. This strategy is detailed in Appendix A.4.
3.2 OBJECT-AWARE INVERSION AND REASSEMBLY
The optimal inversion search be performed for each editing pair, providing us great flexibility in multi-object editing tasks. In short, to solve concept mismatch and poor editing, we disassemble the image add different inversion noise according the optimal step for each region and reassemble the noised regions at the corresponding stage.
Disassembly. From the original and target prompts, we get sequence of editing pairs {(O,, O+) For preparation, we replace the entity Of in P. with Of for each pair in {(O., O,)x., generating a sequence of guided prompts {P*), }, shown in Fig. 3 (a). Then, we feed the original image and the guided prompts into the optimal inversion step search pipeline to obtain the optimal inversion step i for all editing pairs, as illustrated in Fig. 3 (b). Here, each guided prompt is treated as the P. for the optimal inversion step search pipeline. Subsequently, we use the guided prompt for denoising of each editing pair, as depicted in Fig. 3 (c). Moreover, the optimal inversion step searching processes for distinct editing pairs independent. In a multi-GPU scenario, processes for different editing pairs in parallel on multiple GPUs, achieving further acceleration.
run the step searching
The disassembly process segregates the editing processes of different editing pairs, effectively cir- cumventing concept mismatch. Simultaneously, this isolated denoising allows each editing pair to employ the latent from its respective optimal inversion step, thus avoiding poor editing.
Reassembly. In this process, given the inversion step for each editing pair, we edit and integrate the regions into the final result, as illustrated in Fig. (c). We also assign inversion step for the
5

---

Published as a conference paper at ICLR 2024
Origina Promptip? 2) dog aitting noxt to gigft
Origina Iieage
Recssembly Step-10
inverdion stepe24
Col op iininnl
Colie optineal inversion stepe-30
Target PrompttPy)) cat colee...
Editing Pains (dog, on)), (oft, coso)
Guided fromptfcnr) out Editing (dog, cor)
Gurded Premeticale) dag. cake... Editing Pain (gft, colo)
(a) Guided prempts preparction
Guided Promphion))
Pipeline
Guided Prompt(cska)
Original Image
Ritst Fronge (czic)
Cake optinal
Inversion stop-30
End
Timestep-
Cat optinal Inversion step:24
Torgst Tmage
Re-imeenion
Inverse
Search Pipelin
Crop/Splicn
Prompt
(b) Optimal inversion step searching
(c) Object-aware Inversion and Reassembly
Figure 3: Overview of object-aware inversion and reassembly. (a) We create guided prompts for all editing pairs using Po and Pt. (b) For each editing pair, we utilize the optimal inversion step search pipeline
automatically find the optimal inversion step. (c) From each optimal inversion step, we guide the denoising individually using its guided prompt. We crop the denoised latent of the editing regions and splice them with the inverted latent of the non-editing region's the reassembly step. Subsequently, we apply re- revinversion process to the reassembled latent and denoise it guided by Pt.
non non-editing region named reassembly step é,. indicating the editing regions reassemble at this step. Specifically, for the k-th editing region, we start from priise the noised image the optimal inver- sion step ig, and use the guided prompt ph to denoise for i4 - i, steps. This ensures the resulting image I will be at the same sampling step as the non-editing region Iooise To reassemble the regions, we paste each editing result to !io to get the reassembled image 1 at step i,. We found that for most images, setting the reassembly step to 20% of total inversion steps yields satisfactory outcomes. To enhance the fidelity of the editing results and smooth the edges of the editing region, instead of directly denoise from Trr, introduce another noise adding process called re-inversion. Inspired by (Xu et al., 2023; Meng et al., 2023; Song et al., 2023), this process reapplies several inversion steps on the reassembled image 1,. In our experiments, the re-inversion step ire is also set to 20% of the total inversion steps, as we empirically found that it performs well for most situations. Lastly, we use the target prompt P; guide the denoising of the e--iveersinn image Tir+ire facil- itating global information fusion and producing the final editing result. Compared with previous methods, our reassembled latent merges the latents denoised from the optimal inversion steps of all editing pairs, along with the inverted latent from the non-edited region. This combination enables us to produce the best-edited result for each editing pair without compromising the non-editing region.
EXPERIMENTS
We evaluate our method both quantitatively and qualitatively on diverse images collected from the internet and the collection method can be found in Appendix A. The implementation details of our method can found in Appendix A.2. Since single-object editing encompassed in multi- object editing, we mainly present the experimental results on multi-object editing. Detailed results on single-object editing can found in Appendix A.
4.1 MAIN RESULTS
Compared methods. We make comparisons with the state-of-the-art (SOTA) image editing meth- ods, including DiffEdit (Couairon et al., 2022), Null-text Inversion (Mokady et al., 2023), Plug-and- Play (PNP) (Tumanyan al., 2023), and the mask-based stable diffusion inpainting (SDI)'.

---

Published as a conference paper at ICLR 2024
(nezden adf- ballor)
les charck)
Jstree het, caloriul figer)
(beak, Fried epp thapet)
(stortieh, shaffed tay sterfich) slanker!
(neniter, gift bax) (bnybeond, controtior)
Glant Lege plan)) wooden nable)
coochet Wult (vetten,
Orgindl Image
our
ht
Shuble fhasie Trpaietig
Jumy
Nal-ttot Imersce
(d)
(f)
(h)
Figure Qualitative comparisons. From top to bottom: original image, our method (OIR), PNP (Tumanyan et al., 2023), Stable Diffusion Inpainting, DiffEdit (Couairon et al., 2022), Null- text Inversion (Mokady al., 2023). The texts the top the images represent editing pairs.
Evaluation metrics. Following the literatures (Hertz al., 2022; Mokady et al., 2023), we use CLIP (Hessel et al., 2021; Radford et al., 2021) to calculate the alignment of edited image and target prompt. Additionally, we use MS-SSIM (Wang et al., 2003) and LPIPS (Zhang et al., 2018 8a) to evaluate the similarity between the edited image and the original image.
Qualitative comparison. We show some qualitative experimental results in Fig. 4, and additional results can be found in Appendix A. From our experiments, we observe the following: Firstly, SDI and DiffEdit often produce discontinuous boundaries (e.g., the boundary between the tank and the grassland in Fig. 4 (b), and the desk in Fig. (g)). Secondly, feature injection methods (PNP and Null-text Inversion) show better editing results in certain scenarios (e.g., the Lego plant in Fig. (g)). However, they overlook the variations in inversion steps for different editing pairs, leading to poor editing (e.g., the foam in the Fig. 4 (e) and the monitor in Fig. 4 and the water in Fig. (h) are left unedited). Moreover, they face serious concept mismatch, e.g., the color and texture of the "colorful strew hat" in Fig. 4 (c) misled by the tiger's skin. Thirdly, our approach can avoid concept mismatch, since we edit each editing pair individually by disassembly (e.g., the tiger and the colorful hat in Fig. 4 (c)). In addition, reassembly in our method can edit non-editing region, e.g., the shadow in the background changes when the leaves turn into balloon.
Quantitative comparison. We conducted quantitative analyses on our multi-object editing dataset. As illustrated in Tab. 1, we achieve state-of-the-an outcomes on CLIP score, surpassing other meth- ods. Notably, our results show significant improvement over the previous SOTA methods, PNP. Besides, result on MS-SSIM is highly competitive, though it's marginally behind DiffEdit and PNP. It's worth noting that MS- MS-SSIM primarily measures the structural similarity between the output and input images and may not always correlate with the quality of the edit. As the qualitative periments reveal, DiffEdit and PNP occasionally neglects certain objects, leaving them unchanged, which inadvertently boosts the MS-SSIM score.

---

Published as a conference paper at ICLR 2024
4.68%
13.7995
2.18%
CLIP score
MS-SSIM
LPIPS
+Ous
Ours
30.28
0.653
0.329
Nall-tost loversion
Plug-and and-Play
28.45
0.658
0.359
67.70%
SD Inpainting
26.87
0.575
0.398
Diitiinn
DiffFaii
20.98
0.736
0.195
Null-text Inversion
28.02
0.651
0.357
User study results. Users are asked to select the best results in terms of the alignment to target prompts and detail preserva- tion of the input image.
Figure 5:
Table I: Quantitative evaluation. CLIP score measures the alignment image and text, while AS-SSSMM and LPIPS evaluate the similarity between the original and the edited images.
(lighthouse, rocket taking off), optimal inversion step=25
original image
target image
(blue sky, sky with sunset), optimal inversion step=41
Search Mark
bakigg
posn and
(sea, grassland), optimal inversion step=48
166
25
35
40
50 Inversion Step
Figure 6: Visualization of the search metric. The images on the right represent the candidate images obtained using the search metric for each editing pair. In the bottom-left corner, curves are plotted with its x-axis representing the inversion step and the y-axis indicating the search metric S.
User study. We selected 15 images from the collected multi-object dataset for user testing, and compared our OIR with SDI, DifiEdii, PNP. and Null-text Inversion. The study included 58 participants who were asked to consider the alignment to the target prompt and preservation of details of the original image, and then select the most suitable image from set of five randomly arranged images on each occasion. As can be seen from Fig. 5, the image generated by OIR is the favorite 66.7% the participants, demonstrating the superiority of our method. An example of the questionnaire can be found in Appendix A.
4.2 VISUALIZATION OF THE SEARCH METRIC
We visualize the candidate images and their search metric in Fig. to gain a clearer understanding of the trend of the search metric S. It's evident that each editing pair has its own optimal inversion step, with significant variations between them. For instance, the (sea, and grassland) perform optimally between steps 45 and 50. Meanwhile, the (lighthouse, rocket taking off) is most effective around the 25th step, but experience significant background degradation after the 35th step. As shown in the curves in Fig. 6, the optimal inversion step selected by our search metric aligns closely with the optimal editing results, showcasing the efficacy of our approach. In addition, in the curves of each editing pair, observe trend that the search metric first increases and then decreases as the inversion step increases. The reasons are as follows: When the inversion step is small, the background changes slightly, making editing region alignment with the target prompt the dominant
8

---

Published as a conference paper at ICLR 2024
factor in the search metric. As the inversion step grows, the edited result aligns well with the target prompt, amplifying the influence of background consistency in the search metric. More visualization results can be found in Appendix A.11.
Original Image
(house, (house, church)
Ours
w/o re-inversion
w/o re- inversion w/o reassembly step
premature reassembly
(a row of tables, tank)
(leaves, balloon) (wooden framed, self- portrait of Van 6ogh)
(b)
(c)
(d)
Figure 7: Ablations for OIR. The images and texts the far left the original images and their editing pairs. The remaining images represent the results after ablating the OIR, The editing effect within the red box is poor.
4.3 ABLATION STUDY
As shown in Fig. 7, conduct ablation experiments on each module in the OIR. Initially, set the reassembly step to a significantly large value, essentially merging editing pairs at an early stage. As observed in Fig. 7(d), mismatch emerge between different concepts, such as the area designated for the house being overtaken by the trees in the background. Additionally, as depicted in Fig. 7(b), the image edges become notably rough, unrealistic, and contain noise, when re-inversion is omitted. Without re-inversion, different regions are denoised independently, leading to weaker representation of the relationships between them. If neither added, not only the concept mismatch, but the edges are sharp and noisy, as shown in Fig. 7(c).
CONCLUSION AND FUTURE WORK
We have proposed new search metric to seek the optimal inversion step for each editing pair, and this search method represents a new paradigm in image editing. Using search metric, we present an innovative paradigm, dubbed Object-aware Inversion and Reassembly (OIR), for mulit-object image editing. OIR can disentangle the denoising process for each editing pair to prevent concept mismatch or poor editing and reassemble them with the non-editing region while taking into account their interactions. Our OIR can not only deliver remarkable editing results within the editing region but also preserve the non-editing region. achieves impressive performance and quantitative experiments. However, our method requires additional inference time for optimal inversion step search, and the effectiveness of our approach on other generic editing tasks, such video editing, remains be verified. Furthermore, exploring the integration of OIR with other inversion- editing methods also an area worth investigating. We consider addressing these issues in future work.
both qualitative

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Key R&D Program of China (No. 2022ZD0118700 The authors would like thanks Hangzhou City University for accessing its GPU cluster.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 8296-8305, 2020.
Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual- stylegan encoder via iterative refinement. In Proc. IEEE Int. Conf. Comp. Vis., pp. 6711-6720, 2021.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 18208-18218, 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Tran. Graphics, (2(4):1-1,,022,
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and An- tonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 18392-18402, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning-Ho free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.0844 08465, 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv: Comp. Res. Repository, 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 139914413-1 2020.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to-pree to-mopt image editing with cross attention control. arXiv: Comp. Res. Repository, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv: Comp. Res. Repository, 2021.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: review. IEEE Trans. Visualization and Computer Graphics, 26(11):3365-3385, 2019.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for eet-too-mmaee synthesis. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 10124-10134 10134, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. Comp. Vis. Putt. Recogn., pp. 4401-4410, 2019.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proc. IEEE Conf. Comp. Vis. Pan. Recogn., pp. 6007-6017, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv: Comp. Res. Repository, 2023.
10

---

Published as a conference paper at ICLR 2024
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv: Comp. Res. Repository, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv: Comp.
Res. Repository, 2021.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 14297-14306, 2023.
Daiki Miyake, Akihiro lohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv: Comp. Res. Repository, 2023.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 6038-6047, 2023.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text- driven manipulation of stylegan imagery. In Proc. IEEE Int. Conf. Comp. Vis., pp. 2085-2094, October 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arxiv: Comp. Res. Repository, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn., pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nlaa Sutskever. Zero-shot text-to-image generation. In Proc. Int. Conf. Mach. Learn., pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint MrXiv:2204.06/25 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Proc. Advances in Neural Inf. Process. Syst., 35:36479-36494, 2022.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv: Comp. Res. Repository, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 1921-1930, 2023.
Luozhou Wang. Shuai Yang, Shu Liu, and Ying-cong Chen. Not all steps are created equal: Selective diffusion distillation for image manipulation. arXiv preprint arXiv:2307.08448. 2023.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Chrity-Seventhh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1 1398-1402. leee, 2003.
11

---

Published as a conference paper at ICLR 2024
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming- Hsuan Yang. Gan inversion: survey. IEEE Trans. Pattern Anal. Mach. Intell., 45(3):121-3318,, 2022.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong Fine-grained text to image generation with attentional generative adversarial net- works. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 1316-1324, 2018.
He. Attngan:
Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. arXiv: Comp. Res. Repository, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206 10789, 2()):5, 2022.
Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Ko- rtylewski, Christian Theobalt, and Eric Xing. Multimodal image synthesis and editing: The generative Al era. In IEEE Trans. Pattern Anal. Mach. Intell. IEEE, 2023.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proc. IEEE Int. Conf. Comp. Vis., 2017.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proc. Eur. Conf. Comp. Vis., pp. 649-666. Springer, 2016.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 586-595, 2018a.
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with hierarchically-nested adversarial network. In Proc. IEEE Conf. Comp. Vis. Putt. Recogn, 2018b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. IEEE Int. Conf. Comp. Vis., pp. 2223-22233 2017.
12

---

shown in Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DATA COLLECTION
To assess the effectiveness of real-world image editing, we collect two datasets by carefully select- ing images from various reputable websites, namely Pexels 2, Unsplash 3 and 500px 4. We use the first dataset to test the ability for single-object editing of the search metric, including animals, vehicles, food, and more. The second dataset is created to evaluate the method's multi-object editing capabilities. Each photo in this dataset contains two editable objects. We also designed one or more prompts for each image to test the editing effectiveness. The images resize to 512x512 pixels.
A.2 IMPLEMENTATION DETAILS
We use Diffusers implementation of Stable Diffusion v1.4 in our experiments. For DDIM Inver- sion, we used a uniform setting 50 steps. Our method employs the simplest editing paradigm, consisting of first applying DDIM Inversion to transform the original image into noisy latent, and then conducting denoising guided by the target prompt to achieve the desired editing effect. We employ the CLIP base model to compute the CLIP score outlined by (Hessel et al., 2021) for our search metric, and utilize the CLIP large model for quantitative evaluation. Following (Miyake al., 2023), set the negative prompt the original prompt for the denoising process throughout our experiments. We use Grounded-SAN to generate masks. We use our search metric to perform single-object editing and compare with Plug-and-Play (PNP) (Tumanyan et al., 2023), Stable Diffu- sion Inpainting (SDI)², and DiffEdit (Couairon et al., 2022). For multi-object editing, we compare our method to PNP, SDI, DiffEdit, and Null-text Inversion (Mokady et al., 2023), which can directly support multi multi-object editing.
In the single-object editing experiments, the parameters of PNP were kept consistent with the de- fault values specified in the code. DiffEdit utilized the default parameters from the diffusers library. SDI utilized the code from the Diffusers. The random seed is set for all experiments.
In the multi-object editing experiments, PNP can easily be extended to generalized multi-object editing scenarios. For SDI, we consider three approaches to extend it to multi-object scenarios. Method / uses mask to frame out all editing regions and use a target prompt to guide the image editing. In Method 2, different masks are produced for different editing regions. These regions then utilize guided prompts for directed generation. Subsequently, after cropping, the results are seamlessly merged together. In the third approach, we substitute the guided prompt from Method with O specific to each editing pair. Method 3 is used in Fig. 4 because it has the best visual effects. All our experiments are conducted on the GeForce RTX 3090.
A.3 SCHEMATIC COMPARISON
The automatic image selection through search metric is a new image editing paradigm, which is theoretically similar to the feature-injected-based image editing method. We use the most repre- sentative PNP among of 50 steps of DDIM Inversion, PNP will select the latent after 50 steps of inversion, Fig. 8 (a). At this time, latent is the most noisey and has the greatest editability. If we directly denoise the latent, it will severely destroy the layout of the original image. To solve this problem, PNP reduces the editing freedom by injecting features. Compared with PNP, our search metric in Fig. (b) automatically selects the most suitable latent by controlling the number of inversion steps to achieve editing fidality and editability.
image editing methods as an example. In the scenario
"https:/www.peexedt hwtp://wwww.eeee.ccomcccccccll
"hupss//unsplash.coo
'haps://500x.comm
13

---

Published as a conference paper at ICLR 2024
XXX
inject feature
alterable
(o) PNP
(b) Our Search Metric
Figure 8: Left: the process of the feature injection method. Right: the process of our search metric.
A.4
ACCELERATION FOR GENERATING CANDIDATE IMAGES
Unit
Change Letrens Change Timestiss
Pardlil bereise Nethods
X50
X119
Figure 9: The funnel shape represents the denoising process, while the vertical bold lines represent the operations of changing the latent and changing the timestep. (a) Schematic for generating all target images. (b) Our proposed method for implementing parallel generation of all target images. (c) Extending the methodology the 50-step DDIM Inversion.
As shown in Fig. (a), generating candidate images is serial process and there is interde- pendence in different denoise processes. We leverage this characteristic to propose an acceleration method for generating candidate images, illustrated Fig. (b). This method involves equalizing the length of denoise operations and introducing "change latent" and "change timestep" operations the junctions. By denoising all latents simultancously, candidate images to the same speed as generating a picture. An extension of our approach, tailored the context where DDIM Inversion spans 50 steps, shown in Fig. (c).
will change the generation speed of
A.S GENERATING MASK AND VISUAL CLIP FEATURE
We utilize the Grounded-S SAM (Liu et al., 2023; Kirillov et al., 2023) to generate masks of the editing
regions and we will use these masks to compute the CLIP score (Hessel et al., 2021). The detailed process is depicted in Fig. 10 (a), and examples of the segmentation result is presented in Fig. 11. Since only the object features within the mask region are interest, a self-attention mask is applied to restrict the feature extraction of CLIP vision model. The mask is resized match the number of patches in CLIP and then transformed into an attention mask as depicted Fig. 10 (b). Finally, fed into the self-attention of the CLIP vision model for interaction with the original image.
14

---

Published as a conference paper at ICLR 2024
Origin Prompti caT wearlng strow hat is lying the carget
resize generate oetention mosk
Origin Object: cat,
(197x197)
(14x14)
Grounded-SAre
strow hot
CLIP Vision Model
Vilnn fecture
(a) Generate mask
(b) Generate attention mask
Figure 10: Left: the process of generating editing region mask. Right: the process of generating CLIP's self-attention mask through object mask.
(venden fromad sulf- partoo of Vix Gega) Ossnci ballor)
(stroe her coloril strw fost, tiur?
(boal, fried esp fert plane) (sac,, heapor)
(labkes tuna) Poess, shunch)
sterfian) ('sam, chackened Honkort)
(senitor, gift bol (keyboard game controller)
(sker Laga plant) (uunn hond.
Whal-E craduet Wil E) Jerter pil
Figure 11: Segmentation masks for images in Fig.
A.6 COMPARISON OF EDITING SPEED
Ours OIR (Multi-GPU)
Ours OIR
Null-text Inversion 173s
Stable Diflusion
Method
Plug-and-Pie
DiffEdit
(Single-GPU)
Inpainting 6s
Average time cost Maximum GPU
155s
298s
224s
15s
19.75 GB
19.75 GB
22.06 GB
7.31 GB
10.76 GB
13.51 GB
usage
Table 2: Editing speed and maximum GPU usage of different editing methods in multi-object editing.
We evaluate the speed of diverse editing techniques applied to a multi-object dataset using the GeForce RTX 3090, with the results detailed in Table 2. We ignore the time overhead for pre- and post-processing, such model loading and file reading, concentrating primarily computational costs. "Time cost" denotes the expended time on editing an image, and "Maximum GPU usage" represents the peak GPU utilization by single GPU during the editing process. Our OIR implemen- tation uses the acceleration scheme Appendix A.4. In Tab. 2, OIR (Multi-GPU) indicates running OIR two GPUs, while OIR (Single-GPU) runs the same process optimal inversion step for different editing pairs sequentially. We use the default hyperparameters for Null-text Inversion (NTD, Plug-and Pnd- (PNP), and DiffEdit, using their open- codes. We can observe that, although OIR is slower than NTI and PNP on a single GPU, our method ex- cels in editing capability compared to these methods. Additionally, the additional time overhead is
single GPU, searching the
15

---

Published as a conference paper at ICLR 2024
Mask From Attention Map
Mask From Grounded-SAM
rabbit origami rabbit book newspaper
rabbit
book
Editing Result
rabbit
book
Editing Result
dumplings chopsticks >> knife
dumplings
chopsticks
Editing Result
dumplings
chopsticks
Editing Result
Figure 12: Compare the impact of different mask generators in the search metric editing results.
within an acceptable range. Moreover, our method can be accelerated significantly when running on multi-GPUs, outperforming NTI and PNP in speed, where NTI and PNP do not have clear solutions that can be accelerated on multi-GPU due to the temporal dependency between the denoise steps.
A.7 COMPARISON OF DIFFERENT MASK GENERATORS
We compare the influence of different mask generators OIR, shown in Fig. 12. During testing, we employ two types of mask generators. The first approach is the Grounded-SAN. method within the segment model. The second approach involves extracting masks using the attention map from Stable Diffusion, following methods like DiffEdit (Couairon et al., 2022) and MasaCtrl (Cao al., 2023). Specifically, We employ the mask extraction method from DiffEdit, which eliminates the need for introducing an additional model. The first line in Fig. 12 reveals notable accuracy loss in the mask extracted from the attention map compared the one extracted by Grounded-SAN -AM.. Nevertheless, OIR consistently produces excellent editing results with these sub-optimal masks, method across various mask generators. Moreover, as seen from the
indicating the robustness second line in Fig. 12, our method performs well when using the mask extracted from the attention map. Thus, our approach not reliant on the segment model, highlighting robustness in handling different masks and producing plausible editing results.
A.8 THE COMBINATION OF SEARCH METRIC AND OTHER INVERSION -BASED IMAGE EDITING METHODS
Our search metric can be used in conjunction with other inversion-based image editing methods. Here we use the fusion Null-text Inversion (NTI) Mokady et al. (2023) and the search metric as an example. In NTI, the "cross replace step" is crucial hyperparameter that determines the proportion of feature injection. information, while lower value allows for more freedom in editing. In the NTT's open-source code, the "cross replace step" is set to 0.8. There are multiple ways to combine the search metric and NTI. The first approach is fix the inversion step and use the search metric to find the optimal "cross replace step". The second approach is to fix the "cross replace step" and use the search metric find the optimal inversion step. The third approach involves simultancously searching for both the "cross replace step" and inversion steps. Fig. 13 shows the experimental results for the first alternative. From the results in the first row, is clear that the "cross replace step" in the official code fails to transform the wooden house into glass house. By contrast, by exploring the parameters of the search metric, we can achieve improved editing results. As can be seen from the second row in Fig. 13, is evident that the "cross replace step" varies significantly for different editing tasks, making manual adjustment impractical. Therefore, the search metric is highly valuable in this context. Additionally, the search metric be used ensemble learning approach. For example, three editing methods are applied simultaneously, each producing different editing results, the search metric be used select the optimal result the final editing outcome.
higher value for the "cross replace step" retains more the original image
16

---

Published as a conference paper at ICLR 2024
Cross replace step searched search metric
Cross replace step official code
Cross replace step
0.4
0.9
wooden house
glass house
dumplings rice
Figure 13: The combination of Search Metric and Null-text Inverison.
(bird, crochet bird)
(blue sky, sky with sunset)
(cat, tiger)
(water, pile of cofetti)
Number of image
8
10
20
30
40
50
Optimal inversion step
Figure 14: The distribution of optimal inversion steps in multi-object dataset.
A.9 THE DISTRIBUTION OF OPTIMAL INVERSION STEPS IN MULTI-ORJECT DATASET
To determine the distribution of optimal inversion steps images, we use the search metric find the optimal inversion steps for 200 editing pairs in 100 images. The results of these editing pairs are shown in Fig. 14. This figure illustrates the number of optimal inversion steps on the horizontal axis for multi-object images, while the vertical axis represents the number of images corresponding to each optimal inversion step. From Fig. 14, clear that different editing targets require different optimal inversion steps. We notice that larger optimal inversion steps necessary when altering backgrounds or objects with significant shape changes, such as the sky or the ground. Conversely, scenarios with smaller inversion steps typically involve objects and targets with similar shapes.
17

---

Published as a conference paper at ICLR 2024
(two soldiers) (spacecraft,
(cot, car) (jass, gelden cup)
(green house, green castle) (grasey field, snowy field)
(lone full moon) [oun flower, crescent
(monitor, box) (keyboard, game controller)
(TV, playing cortocn picture) (ounch, crochet
Original Image
Ours
NIT P2P
[Groundud-SAM)
(
Figure 15: Our OIR vs. Null-text Inversion with Grounded-SAM.
A.10 OIR Vs. NULL-TEXT INVERSION WITH GROUNDED-SAM,
Null-text Inversion (NTT) (Mokady et al., 2023) is combined with Prompt-to-Prontt (P2P) (Hertz al., 2022) by default, which utilizes the attention map extract masks and improve background preservation, allowing for local edits. We replace the mask generation method with Grounded-SAN, examine whether precise mask extractor would enhance the editing effectiveness of NTL. In Fig. 15, columns b, and с use the word swap with local edit approach from P2P. Due to the different lengths of the original prompt and target prompt, columns e, and f in Fig. 15 utilize prompt refinement with local edit method from P2P. From columns a and d in Fig. 15, we notice that NTI with Grounded- SAM fails to preserve the layout information the original image. From column b, it is evident that NTI cannot effectively address the concept mismatch. From columns e, and f in Fig. 15, it can be seen that NTI fails to overcome the issue of poor editing. The main reason for the poor performance is that NTI does not take into account that different editing pairs for the same image should have distinct optimal inversion steps. What's more, OIR training free, while NTI requires additional training.
A.11 ADDITIONAL RESULTS
CLIP score f MS-SSIM
LPIPS $
Search Metric
27.49
0.719
0.215
Plug-an and-Play (Tumanyan al., 2023)
27.39
0.680
0.293
Stable Diffusion Inpainting
27.36
0.706
0.180
Difflddit (Couairon et al., 2022)
21.23
0.726
0.200
Table 3: Quantitative evaluation for search metric single-object editing. We CLIP (Hessel
et al., 2021; Radford et al., 2021) to calculate the alignment of image and text, and use MS-SSIM (Wang et al., 2003) and LPIPS (Zhang al., 2018a) to evaluate the similarity between the target image and the original image.
We compared the single-object editing capabilities of our search metric with the state-of-the-ar art (SOTA) method, shown in Fig. 16. We have provided quantitative metrics for single-object
18

---

Published as a conference paper at ICLR 2024
(Bonana leaf, broccoll)
(horse, weoden horse)
(Rubik's cube, pine cone)
(cone, birthday cake)
(bird, lomingo)
(buggy, jeep)
(Cot, pando)
(Fried rioe, cake)
Origin image
Ours
PNP
DiffEdit
Stable Diffusion
Inpainting
Figure 16: Qualitative comparison the search metric.
dataset in Tab. 3, and it's evident that our method is comparable to the current SOTA approach. Simultaneously, display numerous OIR results the multi-ohject dataset, depicted in Fig. 17 and Fig. 18. The comparison between OIR and SDI's three methods is shown in Tab. 4. Addition- ally, we have included some search metric visualization experiments, as presented in Fig. 19. The visualization of the optimal results for different editing pairs in Fig. 4 can be seen in Fig. 20. Fig. 21 displays our user study questionnaire form.
CLIP score 1
MS-SSIM 1 LPIPS І
OIR
34.98
0.653
0.329
Stable Diffusion
32.97
0.516
0.420
Inpainting (Method!)
Stable Diffusion
32.02
0.581
0.386
Inpainting (Method2)
Stable Diffusion
32.16
0.575
0.398
Inpainting (Method3)
Table 4: Quantitative evaluation for OIR with Stable Diffusion Inpainting on multi-object editing. We use CLIP (Hessel et al., 2021; Radford et al., 2021) to calculate the alignment of image and text, and use MS-SSIM (Wang et al., 2003) and LPIPS (Zhang et al., 2018a) to evaluate the similarity between the target image and the original image.
19

---

Published as a conference paper at ICLR 2024
[Gat, owl) (dog. golden retriever)
(bes, butteeffy) green leafy basket, porrotChhnistmas cart, cordbcard presents)
(Dog, penguin)
(book, towel)
gold table)
(o)
(dumplings, fry nice) (chopsticks,
(bird, parrot)
(cor, cabriolen)
(ighthouse, rocket taking off) sky, sky surdet))
(tree,
(Wooden house, glass house)
(b)
scoop) (steok, cup of coffee)
(pickup, rusty pickup) (green trees, yellow trees)
(ven, (building, ancient romon architecture)
newspsper) (Wooden floor, marble floor)
(6)
(pancake, cake) (milk, coffes)
(choes, boots) (wooden floor, carpet)
(siganattes, noodles) (spoon, makeup brusk)
(tulips, balloons)
(stormy sky. storry sky)
(lavess, cliport (allipop) (Wooden fromed decorations, portroit the Statue of Liberty)
(lighthouse, rockett) (rocks, ship)
(Cylindrical hay bale, Swiss roll) (ground, lavender)
greas, lovender) (flowers, bunch of roses)
(asphalt road, railway track) (white dky The Starry Night Gogh)
(astronout, Iron Man)
(full moon, bleady eyaball) (water, blazing
(shack, costle) (blue sky, Way sky)
(gloes sphere
rings, UFO)
goiden cheir) (green greas, woeden floor)
(ostronaut, (pink cloud, spaceship)
full moon) (sun flower, crescent moon)
(TV, TV playing contoon picture) (bench, crochet bench)
(9)
(rocks, cakes on picnic mct) (grass of hilltop, lavender field)
(rabbit, fox)
(cot, empty) (corpet, wooden floor)
(cat, crochet cat) (gloss, golden cup)
(doisies, roses)
(h)
(cot, (piano, brick woll)
(skull, Captain (book, wooden box)
head)
(cat, bird) (bean bag chair, pumpkin)
(rabait, origani rebbit) (book, newspoper)
(k)
(m)
Figure 17: Additional qualitative results for OIR. It's evident that our method can edit not only objects but also backgrounds, including the sky and ground, and facilitate style transfer. Examples like (b, k), (b, m), (c. I), (c, m), (d, m) involve background editing, (c, k) encompasses seasonal editing, and (f, j) achieves style transfer.
20

---

Published as a conference paper at ICLR 2024
(morkey. tiger) (stone pillor, weeden stake)
(wooden boot, Lego boet) lake, grean furry blanket)
(wooder boat, white spooscroft) (blue with few stars)
(metol toy car, wooden tey car) (groy ground, gray furry corpet)
(mon black clothes and helmet. motorcycke wwarrng Spider-Non Lego
(toy truck, Lego truck) (orny ground, Lago wooden floor)
(sports car, crochet sports car) (sand dunes, rocks)
(filement, white rose) (book, box)
motorcycle)
(6)
(big tree, broccoli) (proes, green grass field)
(Mountains and trees, mountairs and trees
(grasey field, snow)
plant, Lego plant) (humon hand, floworpot)
(log cabin, crochet house) (forest, yellow forest in autumns)
winter))
(c)
(big tree, broccoli) (gross, green field)
ftwo astronauts, two soldiers) (spacecraft, monument)
(oirplane, Lego airplane) (the through the close shat moon)
(bird, crechet bird) (bronch, gold branch)
(d9)
(parrat, crochet parrat) (branch, gold branch)
(vall-e, wall-e) (water, gress field)
(vall-s, wooden wall-e) (wstee, cluster flowers)
(wall-e, Lego wall-e) (water, table)
(x)
(f)
Figure 18: Additional qualitative results for OIR. It's evident that our method can edit not only objects but also backgrounds, including the sky and ground, and facilitate style transfer. Examples like (a, h), (c. f), (d, f). (d, h), (e, g), (e, h), (e, i) involve background editing. (c, g), (c, encompasses seasonal editing.
21

---

Published as a conference paper at ICLR 2024
(cat, tiger), optimal inversion step=17
Original Image
Target Image
(strow hat, colorful straw hat), optimal inversion step=26
coderia
20
25
Inversion Step
(lighthouse, rocket taking of optimal inversion step=39
Original Image
Target Image
(starfish, stuffed toy starfish), optimal inversion step=34
beanch
staftic toy staten
25
30
35
40
45
50 Inversion Step
(dog, penguin), optimal inversion step=19
Original Image
Target Image
(Christmas cart, cardboard box of presents), optimal inversion step=35
Search Morie
Inversion Step
(straws, lavenders), optimal inversion step=31
Original Image
Target Image
(book, towel), optimal inversion step=33
(wooden table, gold table), optimal inversion step=35
15
20
25
30
35
40 Inversion
Step
Figure 19: Additional visualization results of our search metric.
22

---

Published as a conference paper at ICLR 2024
optinal inversiinn step
optimal inversion stap
optinal inveriion stsp-17
optindl steps
self-portroit Von Gagh
tonk
colorful
plate with fried egg and fork
strow hot
tiger
optinal inversion.
optimal stsp=29
optimal inversion stop=
optinall Inversion $top=30
optinal inversiin step=
optimal e4ep-34
25ep=
crechet wall-e
storfich
bong
table
chequered barket
game controll
Logo plant
conferti
optinal Inversion step:39
ophimal Inversion step=3t
optinol mmversoon step:29
optinel inversion stepe46
Figure 20: The optimal editing results for each editing pair Fig.
23

---

Published as a conference paper at ICLR 2024
*16. The following shows the original image the editing results of image editing methods.
Please select the result that best aligns the target prompt and preserves details of the original image.
Target Prompt:
'vase' into 'teapot'
2. tum 'book' into "plate with fried and fork'
original image:
Figure 21: User study print screen.
24


Reviews:
[{"review_id": "AGJVbAwa85", "reviewer": "Reviewer_xVir", "summary": "The paper proposes object-aware inversion and reassembly for image editing. The motivation is that the inversion steps vary from the editing of different objects. Therefore, we need to choose different inversion steps for different objects. Also, for different objects in one image, we need to merge the editing results. Then the reassembly strategy is introduced. The proposed method achieves state-of-the-art performance.", "questions": "Please see my concerns in the weakness part.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well-organized and easy to follow.\n\n2) It makes sense to use different diffusion steps for different objects when performing the image editing.\n\n3) The paper proposes a reassembly strategy to merge the editing results.\n\n4) The proposed method achieves state-of-the-art performance.", "weaknesses": "1) To determine the optimal inversion steps for image editing, we need to inverse the image to all steps and then edit them accordingly. It is time-consuming and not automatic.\n\n2) The two contributions are more like the engineering stuff. However, admittedly, they do bring a lot of performance gain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q4T84EpVuU", "reviewer": "Reviewer_pgMn", "summary": "This paper discovers that various editing pairs exhibit differing levels of editing complexity. Additionally, it is observed that neglecting the varying difficulty levels of different editing pairs in multi-object editing tasks results in the problems of concept mismatch and poor editing. To address these issues, the paper introduces a novel training-free image editing approach called OIR. This method follows the approach of assembly first and then reassembly. 1) In the assembly strategy, the paper introduces a novel search metric. This metric automatically identifies the optimal inversion step for different editing pairs, enabling automatic control of editing difficulty. This approach allows different editing pairs to undergo separate denoising, preventing concept mismatch issues. Moreover, employing the search metric to find the optimal result represents a new paradigm in single-object editing. 2) In the reassembly strategy, the article suggests merging the editing regions and non-editing region during the reassembly step. This operation takes place in the denoise latent space. The reassembly process incorporates a re-inversion strategy, enhancing the image editing's edge smoothness, improving image editability, and enabling interaction across regions. 3) To assess OIR's capabilities, the authors collect two datasets, which are employed to evaluate both the single-object editing proficiency of the search metric and the multi-object editing capability of OIR. Numerous experimental results indicate that the search metric performs on par with existing state-of-the-art editing methods in single-object editing tasks. Moreover, in multi-object editing tasks, OIR demonstrates strong performance, outperforming the previous SOTA methods", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThis paper identifies a fundamental challenge in multi-object image editing tasks. Previous methods typically treat an entire image as a whole entity during multi-object editing, without considering that editing pairs may have varying levels of editing complexity and therefore require different optimal inversion steps.\n2.\tThe new search metric introduced in this article is simple yet effective. The approach of adding the two evaluation indicators makes sense, and visual verification confirms that the search metric aligns with the editing effect. In single-object editing tasks, employing the search metric yields promising results, comparable to other image editing methods.\n3.\tThe OIR introduced in this article is novel and effective, presenting a new solution for the multi-object editing task. Unlike previous approaches, which treated the entire image as a whole, OIR breaks down the task into editing pairs. The experimental comparisons with other methods are extensive and thorough.\n4.\tThe paper's structure is well-organized and easy to follow. The figures are well-designed, effectively illustrating the ideas and claims presented.", "weaknesses": "1.\tIn [a], it is mentioned that the inversion step can be considered as a fixed hyperparameter. However, the author only presents the results of the optimal inversion steps for example images, without demonstrating the overall distribution trend of the optimal inversion step across the entire dataset. Moreover, what are the distinct characteristics of editing pairs with larger and smaller optimal inversion steps respectively?\n2.\tIn the reassembly strategy of OIR discussed in this paper, both reassembly step and re-inversion are mentioned. The paper indicates that both methods can smooth the edges of images and enable global information interaction. Re-inversion first inverts the spliced latent and then denoises it, essentially increasing the denoise step. Is it possible to replace the operation of re-inversion by increasing the reassembly step?\n[a] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ssTNuOEmXE", "reviewer": "Reviewer_r1AF", "summary": "This paper introduces advancements in text-driven image editing using diffusion-based methods. Existing techniques involve a fixed number of inversion steps to edit images aligned with a target prompt, but the optimal number of steps varies for different editing pairs. The paper introduces a new approach called Object-aware Inversion and Reassembly (OIR) to enable fine-grained, object-level editing. The object-level fine-grained editing is achieved by segmentation from SAM. It uses a search metric to determine the optimal inversion step for each editing pair and combines them for the final edited image, demonstrating superior performance, especially in multi-object editing scenarios.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written, and the idea is clearly demonstrated through figures.\n\n2. Although it is natural to do some grid search on the inversion steps when doing image editing, this paper introduces a systematic and principled way to do the search with a quantitative metric.\n\n3. The idea of working on each object separately and reassemble is a interesting way to do multi-object editing.\n\n4. Results are competitive compared to other inversion based editing techniques.", "weaknesses": "1. The method introduces significant computational overhead. For each input image, it has to run a search for optimal denoising steps for  each edit pair. The search process can be very time consuming. as it requires denoising from each inversion step, plus a metric calculation step. The overall amount of computation for a single image editing task can thus be very large.\n\n2. It relies on SAM to do fine-grained segmentation of the input image to localize the object, and then do localized editing followed by resembling. However, it has some limitations. For example, the segmentation may not work well for small object. In addition, using SAM constrained the edit to object-level, while the applicability of more global change is questionable (e.g., change the style of an image from spring to winter, change the background to Mars, etc.). It may not be able to change the location of an object. \n\n3. Related to above point, the competitive method, such as null-text inversion, does not involve segmentation and localization, therefore it is a bit unfair to compare with them, as in principle they can only benefit from the segmentation and localization. This is independent from the main innovation of this paper, which is searching for optimal inversion step.\n\n4. Overall, it is a method that combines multiple existing component, with significant increase in compute. Therefore, the contribution and significance of the method is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TP7v9888t2", "reviewer": "Reviewer_nVsr", "summary": "This paper achieves high-quality object-level image editing through a simple yet effective method. Authors find that the editing for different parts requires different noising levels used to inverse the given image to latent space. As such, they search for optimal inversion steps for different parts based on a search metric for both the edited and non-edited regions respectively. In order to produce a natural final result, the results of different parts are seamlessly blended, which yields a harmonic and globally coherent editing output. Albeit its simplicity, the proposed method outperforms competitive works when editing real images and shows promise in real usage.", "questions": "The proposed method determines the starting point in the latent space to deviate the semantics of the objects. I think this search strategy is orthogonal to other editing methods that manipulate the latent based on the textural prompt. Hence, I wonder whether the proposed method is compatible with other editing methods, like null inversion. I would like to hear the authors' feedback regarding this.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper identifies an interesting and useful phenomenon that the structure of different objects corresponds to a different level of latent as modeled in the diffusion process. To achieve the optimal trade-off between semantic preservation and text-based editing, the optimal number of inversion steps can be determined based on a quantitative measure. \n\n- The image editing results are indeed impressive, as shown in the main text and the appendix. Both the quantitative and qualitative results demonstrate the advantage over strong prior works. \n\n- Since the denoising process of different image parts is disentangled, the proposed method supports fine-grained multiple object editing, which is not featured in prior works.  \n\n- The parallel denoising strategy effectively reduces the search speed.", "weaknesses": "- One major drawback of the method is the search speed. Since the inversion steps of each editing part should be comprehensively searched, this may pose a challenge when editing for multiple regions. It would be interesting to conduct a coarse-to-fine search strategy for speedup. Moreover, in the paper, it is suggested to report the image editing speed for different methods since some other baselines, like prompt-to-prompt, can edit images in a feed-forward manner. \n\n- Some writing parts can be improved for better clarity. For example, what's the detailed formulation of min-max normalization? It is suggested to better rephrase the term \"concept mismatch\".\n\n- Since the re-inversion step is always set to 20% of the total steps, the non-edit regions will inevitably be affected. Such subtle change is particularly apparent for faces and small objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)