Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing

## Abstract

Abstract Recent advancements in text-guided diffusion models have unlocked powerful image manipulation capabilities, yet balancing reconstruction fidelity and editability for real images remains a significant challenge. In this work, we introduce T ask- O riented D iffusion I nversion ( TODInv ), a novel framework that inverts and edits real images tailored to specific editing tasks by optimizing prompt embeddings within the extended 𝒫 ∗ superscript 𝒫 \mathcal{P}^{*} caligraphic_P start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT space. By leveraging distinct embeddings across different U-Net layers and time steps, TODInv seamlessly integrates inversion and editing through reciprocal optimization, ensuring both high fidelity and precise editability. This hierarchical editing mechanism categorizes tasks into structure, appearance, and global edits, optimizing only those embeddings unaffected by the current editing task. Extensive experiments on benchmark dataset reveal TODInv’s superior performance over existing methods, delivering both quantitative and qualitative enhancements while showcasing its versatility with few-step diffusion model.

## 1 Introduction

Text-guided diffusion models Rombach et al. (); Xue et al. (); Saharia et al. () have achieved significant success in synthesizing realistic images due to their controllability and diversity. Leveraging these effective text-guided diffusion models, numerous works have explored the generative priors of pre-trained diffusion models and successfully applied these capabilities to various downstream tasks Zhao et al. (); Qi et al. (); Wu et al. (); Chen et al. (); Ji et al. (); Baranchuk et al. (), particularly in text-driven image and video editing Wu et al. (); Chai et al. (); Qi et al. (); Tumanyan et al. (); Hertz et al. (); Khachatryan et al. (); Saharia et al. (); Cao et al. (). These technologies enable users to edit images according to their desires via text modification.

When editing a real image $x_{0}$, many text driven image editing methods Hertz et al. (); Cao et al. (); Tumanyan et al. (); Parmar et al. () require to invert $x_{0}$ into the latent space of a pre-trained diffusion model to obtain the corresponding latent codes $\{z_{t}\}_{t=T}^{1}$, which is the inverse process of the diffusion model’s sampling procedure. There are two key aspects to this task: the fidelity of the reconstruction and the editability of the latent codes Garibi et al. (); Pan et al. (). A naive approach to this task is Denoising Diffusion Implicit Models (DDIM) inversion Dhariwal & Nichol (); Song et al. (), which reverses the source image according to the DDIM sampling schedule. However, applying DDIM inversion to text-guided diffusion models often fails due to Classifier Free Guidance (CFG) Ho & Salimans (), which uses conditional text as input and magnifies the approximation error.

To eliminate the approximation error in DDIM inversion, many works Sohl-Dickstein et al. (); Mokady et al. (); Han et al. (); Miyake et al. () align the differences between conditional and unconditional trajectories to ensure that the source image is faithfully reconstructed. In addition to aligning the two trajectories directly, several works reduce the approximation error at each timestep by optimizing the latent codes. Specifically, AIDI Pan et al. (), FPI Meiri et al. (), and ReNoise Garibi et al. () introduce a fixed-point iteration process in each inversion step to obtain accurate latent codes. Furthermore, SPDInv Li et al. () optimizes latent codes directly based on the difference between two adjacent latent codes. Despite the progress made in fidelity reconstruction, the optimized latent codes often exhibit reduced editability Garibi et al. (); Parmar et al. ().

Figure: (a) $\mathcal{P}$ Space
Refer to caption: x1.png

To achieve an ideal balance between reconstruction fidelity and editability, we argue that these two tasks must be intrinsically linked and not treated separately. The inversion process should be highly tailored to the specific editing task at hand. This necessity arises because different edited outputs are modified at varying sampling steps or layers of a diffusion model Patashnik et al. (); Liew et al. (). As a result, for a given real image, it is crucial to obtain distinct optimal latent codes corresponding to each editing output.

Furthermore, we discern that various text-driven image editing tasks can be broadly categorized into three distinct classes: structure editing, appearance editing, and structure-appearance (i.e., global) editing. The modulation of appearance and structure is controlled by different layers within the U-Net architecture during the diffusion process. This leads us to assert that varying levels of editing should correspondingly activate different tiers of text embeddings. These insights motivate the creation of an inversion framework that dynamically integrates edit instructions in a hierarchical manner, thereby ensuring both high fidelity and precise editability.
In this paper, we propose a novel Task-Oriented Diffusion Inversion (TODInv) framework designed to invert and edit real images tailored to specific editing tasks. Our approach focuses on inverting to prompt embeddings in individual layers. This method represents the input real image through a sequence of prompt embeddings, which can be effectively edited in downstream applications. In particular, we optimize the prompt embeddings within the extended prompt embedding space $\mathcal{P}^{*}$ Alaluf et al. (). As illustrated in Fig. 2, unlike the original prompt space $\mathcal{P}$, which shares the same embedding across different time steps and U-Net layers, the $\mathcal{P}^{*}$ space employs distinct embeddings at different layers and time steps. This extended space integrates the disentanglement and expressiveness of time and space, benefiting our inversion in two key aspects:
i) The expressiveness of this latent space facilitates the minimization of inversion errors, significantly enhancing reconstruction accuracy.
ii) Compared to the original $\mathcal{P}$ space, $\mathcal{P}^{*}$ space is more disentangled, which allows for more precise optimization tailored to the specific editing type.

To obtain a faithful reconstruction tailored to the target editing task, we optimize only those prompt embeddings that are agnostic to the current editing, thereby minimizing approximation errors without compromising editability. We conduct extensive experiments on benchmark datasets utilizing various text-driven image editing technologies Hertz et al. (); Cao et al. (); Tumanyan et al. (). As shown in Fig. 1, the experimental results indicate that our method outperforms existing diffusion inversion techniques in both quantitative and qualitative evaluations. Additionally, our method demonstrates strong performance with few-step diffusion models, further showcasing its versatility and effectiveness.

In summary, our contributions are as follows:

- •
We present TODInv, a novel diffusion inversion framework that seamlessly links and jointly optimizes inversion and editing processes, achieving both faithful reconstruction and high editability.
- •
We introduce a task-oriented prompt optimization strategy, categorizing various editing tasks into three types. For each class of editing, we minimize the approximation error by optimizing specific prompt embeddings that are irrelevant to the current editing.
- •
Extensive experiments on benchmark dataset demonstrate the effectiveness of our method over state-of-the-art techniques. Our inversion model also supports few-step diffusion models.

## 2 Related Works

##### Image Editing via Diffusion Models.

Diffusion models Rombach et al. (); Saharia et al. (); Ramesh et al. () have made significant advancements in generating diverse and high-fidelity images guided by text prompts. Leveraging these powerful models, numerous works have harnessed their generative capabilities for text-driven image editing. For instance, Prompt-to-Prompt (P2P) Hertz et al. () manipulates attention modules in Stable Diffusion Rombach et al. () for localized and global edits. Plug-and-Play (PNP) Tumanyan et al. () adjusts spatial features and self-attention modules for fine-grained edits, while Pix2pix-Zero Parmar et al. () retains cross-attention maps for image-to-image translation. Recently, MasaCtrl Cao et al. () has enabled complex non-rigid editing by converting the self-attention module into mutual self-attention. Additionally, several works Wu et al. (); Liu et al. (); Geyer et al. (); Zhang et al. () have extended these methods to video editing. To apply these techniques to real images, inverting the images to the latent space of the diffusion model is a crucial first step.

##### Inversion in Diffusion Models.

Early inversion methods for real image editing focused on Generative Adversarial Networks (GANs) Xu et al. (; ); Creswell & Bharath (); Abdal et al. (; ); Xia et al. (). The advent of diffusion models has shifted attention to diffusion-based inversion methods, which can be categorized into Denoising Diffusion Probabilistic Models (DDPM)-based Huberman-Spiegelglas et al. (); Wu & De la Torre () and Denoising Diffusion Implicit Models (DDIM)-based approaches Garibi et al. (); Dhariwal & Nichol (); Song et al. (); Pan et al. (); Li et al. (); Meiri et al. (). DDPM-based methods leverage the denoising process but require a large number of inversion steps Wu & De la Torre (); Huberman-Spiegelglas et al. (). DDIM-based methods introduce a deterministic DDIM sampler for inversion. However, when CFG is used, DDIM inversion often fails to achieve high-fidelity reconstruction Mokady et al. (). To address these issues, several works Mokady et al. (); Han et al. (); Miyake et al. () align the conditional and unconditional trajectories by optimizing the null text token or the prompt embedding. Concurrently, methods like EDICT Wallace et al. () and BDIA Zhang et al. () introduce invertible networks for inversion. PNPInv Ju et al. () merges differences between reconstruction and editing branches, while NMG Cho et al. () utilizes spatial context from DDIM inversion for faithful editing. Despite these advancements, existing methods still suffer from approximation errors in DDIM inversion, as the process approximates latent $x_{t}$ using $x_{t-1}$. To eliminate these errors, techniques like AIDI Pan et al. (), FPI Meiri et al. (), and ReNoise Garibi et al. () introduce fixed-point iteration processes to optimize latent codes. SPDInv Li et al. () reformulates this iteration as a loss function. However, directly optimizing latent codes often results in reduced editability Garibi et al. (); Parmar et al. ().

In contrast to existing solutions, our task-oriented inversion approach optimizes specific prompt embeddings in an extended prompt space for both inversion and editing, thereby avoiding the trade-off between faithful reconstruction and editability. While our method shares similarities with related works Mokady et al. (); Dong et al. (); Han et al. () in prompt optimization, it distinguishes itself in two key aspects:
1) We optimize prompt embeddings to minimize approximation errors in the text-conditioned trajectory of DDIM inversion, rather than merely aligning null-text and text-conditioned trajectories.
2) Our approach specifically connects the inversion process to the editing tasks by optimizing prompt embeddings in the extended $\mathcal{P}^{*}$ space, focusing on embeddings irrelevant to the current editing task. This ensures high-fidelity reconstruction tailored to specific edits without compromising the ability to perform diverse and precise modifications.

##### Extended Spaces of Diffusion Models.

To better leverage the generative capabilities of diffusion models, several works have analyzed the latent space of these models. Voynov *et al.* Voynov et al. () extended the original prompt space to $\mathcal{P}+$ by using different embeddings for different U-Net layers, disentangling structure and appearance. Prospect Zhang et al. () categorized denoising timesteps into style, content, and layout embeddings. NeTI Alaluf et al. () introduced a space-time space $\mathcal{P}*$ for personalized generation. Our work integrates temporal and layer-wise prompt spaces into a unified space, leveraging its expressiveness and disentanglement to achieve high-fidelity reconstruction and editability in diffusion inversion.

## 3 Methodology

### 3.1 Preliminaries

Figure: Figure 3: Overview of our TODInv. Given a real image, we first encode the image to the initial latent code $z_{0}$ using the encoder of Stable Diffusion. In timestep $t$, we get the latent code $z_{t}$ based on latent code $z_{t-1}$ and fixed source prompt embedding $p$ using Eq. 5, but bring the approximation error. Then we use $z_{t}$ to predict latent code $z^{\prime}_{t}$ and minimize their distance by optimizing specific prompt embeddings according to the edit class. The final latent code $z_{T}$ can be cooperated with various editing methods, with the renewed the target prompts using Eq. 10 (the blue arrows)). *Note that only the structure of “cake” is edited in this example, which belongs to structure edit, We only optimize the appearance-related prompt embeddings (denoted by the colorful boxes without grids). For more detailed illustration on how to select the optimization layers, please see in Fig. 4.*
Refer to caption: x3.png

In this section, we present the background of diffusion models and then analyze the approximation error in DDIM Inversion.

#### 3.1.1 Diffusion Models

Diffusion models aim at mapping the random noise $z_{T}$ to a series latent code $\{z_{t}\}_{t=T}^{1}$, where $T$ is the number of timestep, and finally generate a clean image or latent code $z_{0}$, A diffusion model consists of a training process and a reverse inference process. To train a diffusion model, we add the noise $\epsilon\in\mathcal{N}(0,1)$ to the real image $z_{0}$ to get the latent variable $z_{t}$ using follow equation:

$$ ${z}_{t}=\sqrt{\alpha}_{t}z_{0}+\sqrt{1-\alpha_{t}}\epsilon,$ (1) $$

where $\alpha$ is the hyper-parameter. In a text-guided diffusion model, the text prompt embedding $p$ is conditioned on the network ${\epsilon}_{\theta}$ to predict the noise, and it is trained using the following equation:

$$ $\mathcal{L}_{\mathrm{DM}}=\|{\epsilon}-{\epsilon}_{\theta}({z}_{t},p,t)\|^{2}_ {2}.$ (2) $$

During the inference, the clean image $z_{0}$ can be generated from random noise $z_{T}$ using deterministic DDIM sampler Song et al. () step by step:

$$ $z_{t-1}=\phi_{t}z_{t}+\psi_{t}\epsilon_{\theta}(z_{t},p,t),$ (3) $$

where $\phi_{t}$ and $\psi_{t}$ are sampler parameters, and $\phi_{t}=\frac{\sqrt{\alpha_{t-1}}}{\sqrt{\alpha_{t}}}$, $\psi_{t}=\sqrt{\alpha_{t-1}}\left(\sqrt{\frac{1}{\alpha_{t-1}}-1}-\sqrt{\frac{
1}{\alpha_{t}}-1}\right)$.

#### 3.1.2 DDIM Inversion

Diffusion inversion is a reverse process of sampling, which aims to invert a clean image $z_{0}$ to the noise latent code $z_{T}$. According to Eq. 3, $z_{T}$ can be inverted from $z_{0}$ by following equation iteratively:

$$ $z_{t}=\frac{z_{t-1}-\psi_{t}\epsilon_{\theta}({\color[rgb]{0,0,1}{z_{t}}},p,t) }{\phi_{t}}.$ (4) $$

However, directly computing $z_{t}$ using Eq. 4 is infeasible since the network $\epsilon_{\theta}(\cdot,\cdot)$ needs the $z_{t}$ as input. DDIM inversion assumes that the Ordinary Differential Equation (ODE) process can be reversed in the limit of infinitesimally small steps, and replace $z_{t}$ with $z_{t-1}$ for the noise prediction:

$$ $z_{t}\approx\frac{z_{t-1}-\psi_{t}\epsilon_{\theta}({\color[rgb]{0,0,1}{z_{t-1 }}},p,t)}{\phi_{t}}.$ (5) $$

This approximation error is introduced into every timestep of DDIM inversion, the accumulated errors decrease the reconstruction quality and editing ability Pan et al. (); Meiri et al. (); Li et al. (); Garibi et al. (). Moreover, in the recent few-step diffusion models Luo et al. (; ); Sauer et al. (); Song et al. (), the approximation error between $z_{t-1}$ and $z_{t}$ is significantly large, DDIM inversion suffers worse performance on reconstruction Garibi et al. ().

### 3.2 Approximation Error Minimization

For minimizing the approximation error in the DDIM inversion, existing works Pan et al. (); Meiri et al. (); Garibi et al. (); Li et al. () optimize the latent code $z_{t}$ directly in each timestep. In those works, the fidelity reconstruction can be guaranteed, but compromises the editability.

Instead, we optimize the prompt embeddings, rather than original latent codes. A naive solution is optimizing the prompt embedding in the original prompt space $\mathcal{P}$. In timestep $t$, we first get the latent code $z_{t}$ based on $z_{t-1}$ with DDIM inversion (using Eq. 5), then we take the obtained $z_{t}$ and prompt embedding $p$ to predict another latent code $z_{t}^{\prime}$, and we minimizing the distance between the input and output codes by optimizing prompt embedding $p$. The above description can be represented as:

$$ $z_{t}^{\prime}=\frac{z_{t-1}-\psi_{t}\epsilon_{\theta}({z_{t}},p,t)}{\phi_{t}},$ (6) $$

$$ ${p}^{*}=\arg\min_{p}\|z_{t}^{\prime}-z_{t}\|^{2}_{2}.$ (7) $$

However, optimizing prompt embedding directly has two drawbacks. Firstly, for the original space $\mathcal{P}$, a single text embedding is injected to networks regardless of timesteps and layers of U-Net, the optimization of this shared text embedding limits the minimization of Eq. 7 across different timesteps. Secondly, as indicated by the customized diffusion works Ruiz et al. (); Xu et al. (), the optimized $p^{*}$ also encodes the image context after optimization, leading to the decreased editability.

Figure: Figure 4: We categorize all kinds of editing tasks into three classes and divide different layers of U-Net into structure and appearance layers according to their resolutions. For each kind of editing, we only optimize the prompt embeddings that are irrelevant to this editing.
Refer to caption: x4.png

### 3.3 Task-Oriented Prompt Optimization

For achieving the high fidelity reconstruction meanwhile preserving the editability, we argue that the inversion process should be oriented to the edit task, as a universally optimal latent code adept at both faithful reconstruction and diverse editing tasks is unattainable. We observe various image editing tasks can be broadly categorized into three classes: structure editing (“edit a round yellow cake to square yellow cake”), appearance editing (“edit a round yellow cake to round red cake”), and global editing (“edit a round yellow cake to square red cake”). On the other hand, It’s evidenced that the structure and appearance are modulated by different layers’ prompts Alaluf et al. (); Voynov et al. (). This leads us to assert that varying levels of editing should correspondingly different layers of text embeddings.

In our task-oriented inversion, to avoid embedding the content of specific prompts which decreases the editability after minimizing the approximation error, we only optimize the prompt embeddings that are irrelevant to current editing (see in Fig 4). For example, for the appearance editing, we only update those embeddings related to the structures. As the appearance-related prompt embeddings are kept fixed, the editability will not be decreased. We chose the extended prompt space $\mathcal{P^{*}}$ proposed by Alaluf et al. () for optimization, as it is evidenced to be more expressive and disentangled.

Let $p_{t}^{i}\in\mathcal{P^{*}}$ denotes the prompt embedding injected to the $i$ resolution layer of U-Net at $t$ timestep, we follow Alaluf et al. (); Voynov et al. () that class different layer prompt embeddings into two groups according to the resolution: the structure prompt set in the low-resolution layers: $P_{t}^{str}=[p_{t}^{i},i\in low~{}res~{}layers]$, and the appearance prompt set controls the high-resolution layers: $P_{t}^{app}=[p_{t}^{j},j\in high~{}res~{}layers]$, we first get the latent code $z_{t}^{\prime}$ by replacing $p$ with $[P_{t}^{str},P_{t}^{app}]$ in Eq. 6:

$$ $z_{t}^{\prime}=\frac{z_{t-1}-\psi_{t}\epsilon_{\theta}({z_{t}},[P_{t}^{str},P_ {t}^{app}],t)}{\phi_{t}}.$ (8) $$

Then, for the appearance-related editing, we optimize the irrelevant structure embeddings set $P_{t}^{str}$, and vice versa. For the global editing, we optimize all the prompt embeddings, which can be represented as:

$$ $\begin{split}&{P}^{*}_{t}=\begin{cases}\arg\displaystyle\min_{{P}{{}_{t}^{app} }}\|z_{t}-z_{t}^{\prime}\|^{2}_{2}&{if}~{}{\texttt{structure~{}editing}};\\ \arg\displaystyle\min_{{P}{{}_{t}^{str}}}\|z_{t}-z_{t}^{\prime}\|^{2}_{2}&{ elif}~{}{\texttt{appearance~{}editing}};\\ \arg\displaystyle\min_{{{P}{{}_{t}^{str}},{P}{{}_{t}^{app}}}}\|z_{t}-z_{t}^{ \prime}\|^{2}_{2}&{else}~{}{\texttt{global~{}editing}}.\\ \end{cases}\end{split}$ (9) $$

We follow Li et al. (); Dong et al. () that set the maximum optimization steps as $K$ in each timestep, meanwhile, we also set a threshold $\delta$ to control the termination of the optimization process. By feeding the latent code $z_{t}$ with the optimized prompt embeddings ${P}^{*}_{t}$ to the U-Net, with the DDIM sampler, the original image can be reconstructed faithfully. More importantly, with task-oriented optimization, the editability will not be decreased.
If the same image undergoes multiple types of edits during iterative editing, we choose global editing for optimization. This is because applying different edit categories requires optimizing prompt embeddings across all layers, similar to the global editing category.

During the editing, we leverage the difference between the original and optimized embeddings on the target prompt ${P}^{target}_{t}$, that is:

$$ $\tilde{P}{{}^{target}_{t}}={P}^{*}_{t}-{P}{{}_{t}}+{P}^{target}_{t},$ (10) $$

where $\tilde{P}^{target}_{t}$ is the renewed target prompt embedding. Incorporated with various text-driven image editing methods Cao et al. (); Hertz et al. (); Tumanyan et al. (), we can edit the real image with target prompt.

## 4 Experiments

### 4.1 Experimental Settings

**Table 1: Qualitative comparisons with related works using various text-guided editing methods.**
| Method | Structure | Background Preservation | CLIP Similarity | Times(s) $\downarrow$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Inverse | Editing | Distance${}_{{}^{\times 10^{3}}}$ $\downarrow$ | PSNR $\uparrow$ | LPIPS${}_{{}^{\times 10^{3}}}$ $\downarrow$ | MSE${}_{{}^{\times 10^{4}}}$ $\downarrow$ | SSIM${}_{{}^{\times 10^{2}}}$ $\uparrow$ | Whole $\uparrow$ | Edited $\uparrow$ |  |
| DDIM | P2P | 69.43 | 17.87 | 208.80 | 219.88 | 71.14 | 25.01 | 22.44 | 11.55 |
| NTI | P2P | 13.44 | 27.03 | 60.67 | 35.86 | 84.11 | 24.75 | 21.86 | 137.54 |
| NPI | P2P | 16.17 | 26.21 | 69.01 | 39.73 | 83.40 | 24.61 | 21.87 | 11.75 |
| StyleD | P2P | 11.65 | 26.05 | 66.10 | 38.63 | 83.42 | 24.78 | 21.72 | 382.98 |
| AIDI | P2P | 12.16 | 27.01 | 56.39 | 36.90 | 84.27 | 24.92 | 20.86 | 87.21 |
| FPI | P2P | 14.71 | 26.61 | 61.97 | 37.64 | 83.52 | 23.93 | 21.35 | 11.75 |
| NMG | P2P | 26.64 | 25.38 | 88.31 | 112.77 | 81.73 | 24.90 | 22.16 | 16.71 |
| ProxEdit | P2P | 8.80 | 28.31 | 44.13 | 25.72 | 85.74 | 24.15 | 21.36 | 11.75 |
| PNPInv | P2P | 11.65 | 27.22 | 54.55 | 32.86 | 84.76 | 25.02 | 22.10 | 19.94 |
| SPDInv | P2P | 8.81 | 28.60 | 36.01 | 24.54 | 86.23 | 25.26 | - | 27.04 |
| TODInv | P2P | 8.37 | 28.39 | 39.86 | 25.71 | 86.04 | 25.47 | 21.91 | 21.02 |
| DDIM | MasaCtrl | 28.38 | 22.17 | 106.62 | 86.97 | 79.67 | 23.96 | 21.16 | 11.55 |
| AIDI | MasaCtrl | 55.93 | 19.25 | 177.57 | 178.13 | 75.58 | 24.01 | 21.07 | 87.21 |
| NMG | MasaCtrl | 40.54 | 20.35 | 127.85 | 135.17 | 77.52 | 24.56 | 21.33 | 16.71 |
| ProxEdit | MasaCtrl | 21.28 | 23.81 | 85.52 | 66.47 | 81.62 | 23.60 | 20.94 | 11.75 |
| PNPInv | MasaCtrl | 24.70 | 22.64 | 87.94 | 81.09 | 81.33 | 24.38 | 21.35 | 19.94 |
| SPDInv | MasaCtrl | 20.48 | 24.12 | 71.74 | 64.77 | 82.54 | 24.61 | - | 27.04 |
| TODInv | MasaCtrl | 19.39 | 24.36 | 70.17 | 62.27 | 82.95 | 24.74 | 21.20 | 21.02 |
| DDIM | PNP | 28.22 | 22.28 | 113.33 | 83.51 | 79.00 | 25.41 | 22.55 | 11.55 |
| AIDI | PNP | 25.36 | 23.11 | 98.10 | 78.19 | 80.57 | 25.03 | 22.70 | 87.21 |
| PNPInv | PNP | 24.29 | 22.46 | 106.06 | 80.45 | 79.68 | 25.41 | 22.62 | 19.94 |
| SPDInv | PNP | 15.58 | 26.72 | 91.55 | 34.69 | 82.04 | 25.14 | - | 27.04 |
| TODInv | PNP | 21.06 | 25.13 | 78.49 | 50.16 | 82.83 | 26.08 | 22.50 | 21.02 |
| DDIM | P2P-Zero | 61.68 | 20.44 | 172.22 | 144.12 | 74.67 | 22.80 | 20.54 | 11.55 |
| PNPInv | P2P-Zero | 49.22 | 21.53 | 138.98 | 127.32 | 77.05 | 23.31 | 21.05 | 19.94 |
| TODInv | P2P-Zero | 49.86 | 21.34 | 139.47 | 134.66 | 76.91 | 24.19 | 21.15 | 21.02 |
| DDIM^† | ReNoise | 216.17 | 14.52 | 319.53 | 464.16 | 54.30 | 21.17 | 18.38 | 0.56 |
| ReNoise^† | ReNoise | 107.56 | 15.60 | 271.39 | 704.96 | 62.48 | 25.64 | 23.64 | 2.56 |
| TODInv^† | ReNoise | 86.91 | 17.81 | 194.00 | 224.86 | 65.15 | 26.36 | 23.83 | 4.02 |

Dataset. To evaluate the effectiveness of our hierarchical inversion, we conduct experiments on the PIE-Bench dataset proposed by PNPInv Ju et al. (), which consists of 700 images with 9 editing types. Each image is annotated with the source and target prompts. Meanwhile, this dataset also provides the editing region masks for evaluation. For more detailed information about this dataset, please refer to Ju et al. ().

Evaluation Metrics. We follow PNPInv Ju et al. () which uses several metrics to evaluate our method. We first use the Structure Distance assessed by DINO score Caron et al. () to evaluate the structure distance between original and edited images. Note that this metric cannot be used to evaluate structural edits, as neither higher nor lower values effectively reflect the desired changes. However, we follow the official evaluation proposed by Ju et al. (), which adopts a “lower is better” approach for the entire dataset. We also introduce several metrics to evaluating the background preservation, which includes PSNR, LPIPS Zhang et al. (), MSE, and SSIM Wang et al. (). Those metrics are calculated on the unedited regions, which are defined by the PIE-Bench dataset. Additionally, we introduce CLIP Similarity Wu et al. () to evaluate the text-image consistency between edited images and corresponding target editing text prompts. We follow PNPInv Ju et al. () that evaluates CLIP similarity both on the whole image and edited regions, which is denoted by Whole and Edited. At last, we introduce the Inference Times to evaluate different methods’ inversion time costs on a single image.

Image Editing Methods. We incooperate with various inversion methods with four text-guided image editing methods, including P2P Hertz et al. (), MasaCtrl Cao et al. (), PNP Tumanyan et al. (), and Pixel-Zero Parmar et al. (). Note that not all inversion method provides the source code with MasaCtrl, PNP, and Pixel-Zero editing, we only compare all methods with P2P editing. Since there is no editing method available for the few-step diffusion models, we follow ReNoise which edits the images by replacing the target word directly.

Figure: (a) Source
Refer to caption: extracted/5810451/figures/comparison/Source/821000000005.jpg

### 4.2 Implementation Details

We implement the proposed method in PyTorch on a PC with Nvidia GeForce RTX 3090. We use Stable Diffusion V1.4 as our main text-guided diffusion model and set the CFG scale as 7.5. We use the AdamW optimizer Loshchilov & Hutter () with the learning rate is set to be 0.001. We categorize 9 editing types in PIE-Bench dataset into three classes. Particularly, the structure editing contains Add Object, Delete Object, Change Content, and Change Pose. The appearance editing contains Change Color, Change Material, and Change Style, and the global editing only contains Change Background. Additionally, the U-Net of diffusion model has 4 resolution layer scales: $64\times 64$, $32\times 32$, $16\times 16$, and $8\times 8$. Inspired by Voynov et al. (), we take the resolutions of $64\times 64$ and $32\times 32$ as appearance layers, and $16\times 16$, $8\times 8$ as structure layers. We set the maximization optimization steps *K=10*, and follow Mokady et al. (); Li et al. () set threshold $\delta$ as $5e^{-6}$.

### 4.3 Quantitative Comparison

We present the quantitative comparisons with state-of-the-art methods based on various text-guided image editing methods in Tab. 1, we can see that our TODInv outperforms competitors with various editing techniques on most of the evaluation metrics. SPDInv is beyond our method on some reconstruction metrics, but it has a worse editability. As discussed in Sec.3.2, that is because it optimizes the latent code directly for the faithful reconstruction, but ignores the important editing task, the same conclusion also can be drawn from Fig. 1 and Fig. 5, as it always failed on image editing. Thanks to our task-oriented prompt optimization, our method achieves faithful reconstruction and high editability performance. On the other hand, our method is more efficient than optimization works, because we optimize prompt embedding in the expressive $\mathcal{P^{*}}$ space, which is easier for optimization.

### 4.4 Qualitative Comparison

The qualitative comparison with various inversion methods based on P2P Hertz et al. () edit can be seen in Fig. 1 and Fig. 5. We can see that the edited images obtained by DDIM always present an inconsistent background or structure with the source images, as pointed out by NTI Dong et al. (), that is aroused by the CFG used in the sampling process.

Besides, all methods fail to replace the “Jacket” with “Blouse” in $1_{st}$ sample of Fig. 1 except ours, which indicates the effectiveness of our model in object replacement. The same conclusion also can be drawn from the $1_{st}$ sample of Fig. 5, as none of competitors can remove the “Snow” on the fox’s face. By disentangling the structure and appearance editing in the $\mathcal{P^{*}}$ space, our method is also skilled at changing the style of images, such as stylizing real images into “Watercolor”. We notice that SPDInv, AIDI, and FPI fail to replace the “Bread” with “Meat” in the $2_{nd}$ sample of Fig. 5, that is because all of them optimize the latent code for the faithful reconstruction, but reduces the editability. By minimizing the approximation error in each inversion timestep with a specific layer’s prompt optimization, our method not only preserves the source background and structure but also supports various edits. As shown in Fig. 11, our method presents excellent editability incooperated with PNP editing.
For more qualitative comparisons using other editing methods, please see the supplementary material.

### 4.5 Ablation Studies

**Table 2: Qualitative comparisons with various variants using P2P editing.**
| Variant | Structure | Background Preservation | CLIP Similarity | Times(s) $\downarrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Distance${}_{{}^{\times 10^{3}}}$ $\downarrow$ | PSNR $\uparrow$ | LPIPS${}_{{}^{\times 10^{3}}}$ $\downarrow$ | MSE${}_{{}^{\times 10^{4}}}$ $\downarrow$ | SSIM${}_{{}^{\times 10^{2}}}$ $\uparrow$ | Whole $\uparrow$ | Edited $\uparrow$ |  |  |
| *Opti.* in $\mathcal{P}$ | 36.16 | 21.62 | 121.03 | 103.34 | 78.10 | 25.51 | 22.28 | 21.02 |
| *Opti.* in $\mathcal{P}_{t}$ | 35.93 | 21.71 | 120.47 | 102.25 | 78.15 | 25.49 | 22.38 | 21.02 |
| *Opti.* in $\mathcal{P}$+ | 36.40 | 21.63 | 120.92 | 102.61 | 78.12 | 25.57 | 22.33 | 21.02 |
| *T=50, K=25* | 8.32 | 28.36 | 40.04 | 25.68 | 85.92 | 25.47 | 21.89 | 29.04 |
| *T=50, K=50* | 8.29 | 28.37 | 39.93 | 25.66 | 85.93 | 25.45 | 21.89 | 45.04 |
| *T=10, K=10* | 25.01 | 23.89 | 85.51 | 65.81 | 81.28 | 25.64 | 22.02 | 6.79 |
| *T=100, K=10* | 35.10 | 21.70 | 119.20 | 102.36 | 78.29 | 25.61 | 22.26 | 41.23 |
| *w/o* TOPO | 8.55 | 28.18 | 41.24 | 26.48 | 85.80 | 24.46 | 20.14 | 21.02 |
| TODInv | 8.37 | 28.39 | 39.86 | 25.71 | 86.04 | 25.47 | 21.91 | 21.02 |

In this section, we conduct an ablation experiment to analyze different choices in our TODInv. We first analyze the effectiveness of optimization in extended prompt space $\mathcal{P}^{*}$. Particularly, we develop three variants: 1) *Opti.* in $\mathcal{P}$, we optimize the prompt embedding in the original prompt embedding space $\mathcal{P}$, in which all timesteps and layers of U-Net share the same optimized prompt embedding. 2) *Opti.* in $\mathcal{P}_{t}$, we optimize the prompt only in different timestep, in which all layers of U-Net share the same optimized prompt embedding. 3) *Opti.* in $\mathcal{P}$+, we optimize the prompt only in different layers of U-Net, and all timesteps share the same optimized embeddings. We also conduct an ablation study to investigate the effect of different sampling steps $T$ and optimization steps $K$. We develop two variants with different sampling steps *T*, *T=10*, and *T=100*, with the default optimization steps *K=10*, and develop another two variants with different optimization steps *K=25*, *K=50* with *T=50*. Additionally, for evaluating the effectiveness of our Task-Oriented Optimization by proposing variant *w/o* Task-Oriented Prompt Optimization (TOPO) that optimizes all layers of U-Net regardless of the editing types. We conduct above ablation experiment using P2P editing on the PIE-Bench dataset.

Figure: (a) Source
Refer to caption: extracted/5810451/figures/comparison/Source/SDXLT/000000000009.jpg

The quantitative comparison of various variants is presented in Tab. 2. The variants *Opti.* in $\mathcal{P}$, *Opti.* in $\mathcal{P}{t}$, and *Opti.* in $\mathcal{P}$+ demonstrate worse performance in both structure distance and reconstruction. This suggests that optimizing prompt embeddings in these three spaces does not guarantee faithful reconstruction. Additionally, these variants show higher editability (CLIP Similarity) compared to our TODInv, as the edited images, without the constraint of source images, have more freedom to generate content according to the target prompt. In comparison, our final model, TODInv, outperforms variants *T=50, K=25* and *T=50, K=50* across all metrics, although the latter variants require more processing time. The expressiveness of the $\mathcal{P}^{*}$ space facilitates more effective minimization of approximation error, and 10 steps are sufficient for this process. Furthermore, both variants *T=10, K=10* and *T=100, K=10* exhibit poorer reconstruction performance. Consequently, we adhere to existing work by setting *T=50*.

Compared with variant *w/o* TOPO, our final method gains the improvement in editability and reconstruction. Our task-oriented prompt optimization reduces the approximation error by optimizing prompt embeddings that are irrelevant to current editing, and achieves better editability without influencing the reconstruction, which evidences the effectiveness of our task-oriented strategy. For the qualitative comparison of various variants, and quantitative comparison on different editing types, please see in the Appendix.

### 4.6 Extension on Few-step Diffusion Model

Besides the Stable Diffusion, We also extend our method on a few-step diffusion model, SDXL-Turbo Sauer et al. (). We set 4 inference steps for this model, and the optimization steps $K$ is
set to be 10. We compare our method with DDIM inversion, and ReNoise Garibi et al. () in the bottom rows of Tab. 1.
Here we set ReNoise with the DDIM sampler for the fair comparison. We can see that our method outperforms DDIM and ReNoise both on the background preservation and CLIP similarity, with the similar inference time cost with ReNoise, which demonstrates our generalization ability on few-step diffusion model. The qualitative comparison are shown in Fig. 6, our method captures the source structure effectively.

## 5 Conclusion and Limitation

In this paper, we present TODInv, a framework that inverts and edits a real image using diffusion models tailored to specific editing tasks. We categorize various editing tasks into three types, for each kind of editing, we minimize the approximation error by optimizing specific prompt embeddings that are irrelevant to the current editing, achieving both faithful reconstruction and high editability. We conducted experiments on Stable Diffusion and SDXL-Turbo models, demonstrating the effectiveness of our TODInv over state-of-the-art methods. The primary limitation of TODInv is that it requires determining the editing types prior to inversion. However, this can be addressed by using a large language model to easily determine the types. Please refer to the Appendix for detailed instructions.

## Appendix A Appendix

### A.1 Qualitative Comparison of different variants

We present a qualitative comparison of different variants in Fig. 7. The images edited by the variants *Opti.* in $\mathcal{P}$, *Opti.* in $\mathcal{P}_{t}$, and *Opti.* in $\mathcal{P}$+ show inferior results. These variants fail to preserve necessary information from the source images. In contrast, TODInv not only edits the images according to the target prompt but also maintains the unchanged parts of the image. This demonstrates the effectiveness of optimization in the $\mathcal{P}^{*}$ space, which preserves source information and allows for effective editing. Variants *T=50, K=25* and *T=50, K=50* yield results similar to TODInv, indicating that additional optimization steps are unnecessary for TODInv.

In comparison, the variant *w/o* TOPO shows structural deformation in the last sample of Fig. 7 and background perturbation in the $2_{nd}$ and $4_{th}$ samples. With our task-oriented prompt optimization strategy, we only optimize prompt embeddings relevant to the current editing type. This approach not only reconstructs the unedited regions but also preserves editability.

Figure: (a) Source
Refer to caption: extracted/5810451/figures/ablation/Source/000000000005.jpg

### A.2 Analysis on Task-Oriented Prompt Optimization Strategy

To demonstrate the effectiveness of our task-oriented prompt optimization strategy, we present a quantitative comparison across different editing types. We evaluate variants *w/o* TOPO for appearance editing and *w/o* TOPO for structure editing. Additionally, we present the results of reversing the editing type (TODInv-Reverse), wherein appearance editing is applied to samples originally intended for structure editing and vice versa. As discussed in Sec. 4.1, the Structure Distance metric is not suitable for evaluating whether the images are correctly edited; therefore, we exclude this metric from the evaluation of structure editing.

The quantitative comparison is shown in Tab. 3. All variants achieve similar performance in background preservation metrics for both appearance and structure editing, as they are all optimized in the expressive $\mathcal{P}^{*}$ space. Our strategy optimizes prompt embeddings that are independent of the editing type, which enhances editability. Consequently, TODInv-Reverse exhibits poorer performance in CLIP similarity metrics for both appearance and structure editing compared to TODInv, which achieves the best CLIP similarity performance.

**Table 3: Qualitative comparisons with various variants on different editing types.**
| Variant | Editing Type | Structure | Background Preservation | CLIP Similarity |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Distance${}_{{}^{\times 10^{3}}}$ $\downarrow$ | PSNR $\uparrow$ | LPIPS${}_{{}^{\times 10^{3}}}$ $\downarrow$ | MSE${}_{{}^{\times 10^{4}}}$ $\downarrow$ | SSIM${}_{{}^{\times 10^{2}}}$ $\uparrow$ | Whole $\uparrow$ | Edited $\uparrow$ |  |  |
| *w/o* TOPO | Appearance | 8.87 | 29.16 | 38.71 | 26.75 | 86.85 | 25.44 | 23.41 |
| TODInv-Reverse | Appearance | 9.18 | 29.00 | 38.95 | 27.79 | 86.97 | 25.29 | 23.09 |
| TODInv | Appearance | 9.17 | 29.07 | 38.83 | 27.65 | 86.94 | 26.23 | 24.04 |
| *w/o* TOPO | Structure | - | 28.31 | 44.22 | 25.84 | 84.70 | 24.23 | 19.62 |
| TODInv-Reverse | Structure | - | 27.66 | 44.85 | 25.62 | 84.74 | 24.18 | 19.53 |
| TODInv | Structure | - | 28.01 | 42.49 | 24.39 | 85.07 | 25.24 | 20.63 |

We also present the qualitative comparison in Fig. 8, showing that variant *w/o* TOPO and TODInv-Reverse easily present the structure deformation. As shown in the red circle in $1_{st}$ sample of Fig. 8, variant *w/o* TOPO and TODInv-Reverse present the undesired arms in the edited images and modify the view of lions in $2_{nd}$ sample. In $3_{rd}$ sample, both variant *w/o* TOPO and TODInv-Reverse fail to preserve the facial features of source faces, and variant TODInv-Reverse also modifies the “legs” of children. In $4_{th}$ sample, neither variant *w/o* TOPO and TODInv-Reverse failed to remove the “flower”, which further demonstrates the effectiveness of our task-oriented prompt optimization strategy.

Figure: (a) Source
Refer to caption: extracted/5810451/figures/ablation/Source/122000000004.jpg

### A.3 Quantitative Comparison on different editing categories

We present the quantitative comparison on different editing categories date in Tab. 4, Tab. 5, and Tab. 6. Here we use the edited results of other methods provided by PNP’s re-implementation Ju (). From Tab. 4 we can see that our TODInv outperforms other methods with P2P, MasaCtrl, and PNP editing methods on appearance editing categories on all metrics, especially on the structure preservation, our method outperforms other methods with a large step, that demonstrates the effectiveness of our TOPO strategy, by only optimizing the irrelevant layers with appearance editing, our TODInv preserves the structures information of original images effectively.

The quantitative comparison of the images with structure editing category can be seen in Tab. 5. Our TODInv outperforms other methods on all metrics with most editing methods, except with the P2P-Zero editing on background preservation, that is because P2P-Zero is proposed for image translation but not prompt-driven image editing. Compared with P2P, PNP, and MasaCtrl, DDIM and PNPInv inversion methods also receive worse performance on background preservation.

At last, the quantitative comparison of the images with global editing category can be seen in Tab. 6. Our TODInv also goes beyond other methods on most metrics.

**Table 4: Qualitative comparisons on appearance editing category with related works using various text-guided editing methods.**
| Method | Structure | Background Preservation | CLIP Similarity |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Inverse | Editing | Editing Type | Distance${}_{{}^{\times 10^{3}}}$ $\downarrow$ | PSNR $\uparrow$ | LPIPS${}_{{}^{\times 10^{3}}}$ $\downarrow$ | MSE${}_{{}^{\times 10^{4}}}$ $\downarrow$ | SSIM${}_{{}^{\times 10^{2}}}$ $\uparrow$ | Whole $\uparrow$ | Edited $\uparrow$ |
| DDIM | P2P | Appearance | 67.93 | 17.97 | 203.70 | 214.33 | 72.71 | 25.21 | 23.75 |
| NTI | P2P | Appearance | 14.45 | 28.10 | 55.73 | 32.46 | 85.64 | 25.77 | 24.06 |
| NPI | P2P | Appearance | 18.63 | 26.78 | 66.08 | 39.24 | 84.70 | 25.43 | 23.80 |
| StyleD | P2P | Appearance | 12.11 | 26.76 | 63.86 | 36.88 | 84.81 | 25.27 | 23.40 |
| PNPInv | P2P | Appearance | 12.39 | 28.53 | 48.22 | 27.65 | 86.39 | 25.69 | 23.93 |
| TODInv | P2P | Appearance | 9.17 | 29.07 | 38.83 | 27.65 | 86.94 | 26.23 | 24.04 |
| DDIM | MasaCtrl | Appearance | 29.09 | 22.38 | 101.20 | 84.88 | 81.03 | 24.00 | 22.20 |
| PNPInv | MasaCtrl | Appearance | 24.49 | 22.95 | 84.23 | 79.83 | 82.50 | 24.37 | 22.55 |
| TODInv | MasaCtrl | Appearance | 18.66 | 24.66 | 66.94 | 60.81 | 84.30 | 24.66 | 22.55 |
| DDIM | PNP | Appearance | 30.91 | 22.61 | 110.11 | 76.64 | 80.18 | 26.2 | 24.49 |
| PNPInv | PNP | Appearance | 26.40 | 22.89 | 104.77 | 73.82 | 81.02 | 26.21 | 24.62 |
| TODInv | PNP | Appearance | 24.22 | 25.31 | 77.87 | 54.32 | 84.13 | 27.50 | 25.43 |
| DDIM | P2P-Zero | Appearance | 74.20 | 20.21 | 169.57 | 147.82 | 76.12 | 22.95 | 21.76 |
| PNPInv | P2P-Zero | Appearance | 65.51 | 21.30 | 137.77 | 134.84 | 78.45 | 23.53 | 22.18 |
| TODInv | P2P-Zero | Appearance | 62.70 | 21.05 | 138.70 | 137.10 | 78.60 | 24.39 | 22.62 |

**Table 5: Qualitative comparisons on structure editing category with related works using various text-guided editing methods.**
| Method | Background Preservation | CLIP Similarity |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Inverse | Editing | Editing Type | PSNR $\uparrow$ | LPIPS${}_{{}^{\times 10^{3}}}$ $\downarrow$ | MSE${}_{{}^{\times 10^{4}}}$ $\downarrow$ | SSIM${}_{{}^{\times 10^{2}}}$ $\uparrow$ | Whole $\uparrow$ | Edited $\uparrow$ |
| DDIM | P2P | Structure | 17.27 | 230.19 | 237.38 | 68.45 | 24.98 | 21.33 |
| NTI | P2P | Structure | 26.30 | 69.64 | 38.70 | 82.43 | 24.23 | 20.44 |
| NPI | P2P | Structure | 25.66 | 76.98 | 41.20 | 81.82 | 24.15 | 20.54 |
| StyleD | P2P | Structure | 25.53 | 72.62 | 39.44 | 81.99 | 24.57 | 20.65 |
| PNPInv | P2P | Structure | 26.41 | 61.44 | 35.57 | 83.24 | 24.72 | 20.94 |
| TODInv | P2P | Structure | 28.01 | 42.49 | 24.39 | 85.07 | 25.24 | 20.63 |
| DDIM | MasaCtrl | Structure | 21.51 | 118.38 | 95.02 | 77.73 | 24.29 | 20.49 |
| PNPInv | MasaCtrl | Structure | 21.99 | 97.51 | 88.16 | 79.62 | 24.76 | 20.65 |
| TODInv | MasaCtrl | Structure | 23.82 | 77.82 | 66.50 | 81.36 | 25.15 | 22.49 |
| DDIM | PNP | Structure | 21.73 | 125.06 | 90.09 | 77.12 | 25.12 | 21.25 |
| PNPInv | PNP | Structure | 21.86 | 116.15 | 86.30 | 77.83 | 25.15 | 21.35 |
| TODInv | PNP | Structure | 25.04 | 82.28 | 47.75 | 81.55 | 25.48 | 20.76 |
| DDIM | P2P-Zero | Structure | 19.88 | 193.89 | 156.54 | 71.94 | 22.54 | 19.49 |
| PNPInv | P2P-Zero | Structure | 21.00 | 156.94 | 136.81 | 74.53 | 22.95 | 19.98 |
| TODInv | P2P-Zero | Structure | 20.80 | 158.12 | 150.68 | 74.27 | 23.90 | 20.03 |

**Table 6: Qualitative comparisons on global editing category with related works using various text-guided editing methods.**
| Method | Structure | Background Preservation | CLIP Similarity |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Inverse | Editing | Editing Type | Distance${}_{{}^{\times 10^{3}}}$ $\downarrow$ | PSNR $\uparrow$ | LPIPS${}_{{}^{\times 10^{3}}}$ $\downarrow$ | MSE${}_{{}^{\times 10^{4}}}$ $\downarrow$ | SSIM${}_{{}^{\times 10^{2}}}$ $\uparrow$ | Whole $\uparrow$ | Edited $\uparrow$ |
| DDIM | P2P | Global | 66.97 | 19.12 | 165.37 | 185.68 | 75.70 | 24.78 | 23.02 |
| NTI | P2P | Global | 16.56 | 27.50 | 48.43 | 34.37 | 86.10 | 24.40 | 21.69 |
| NPI | P2P | Global | 17.80 | 26.93 | 53.82 | 36.87 | 85.73 | 24.42 | 21.98 |
| StyleD | P2P | Global | 14.44 | 26.54 | 53.52 | 38.47 | 85.33 | 24.49 | 21.64 |
| PNPInv | P2P | Global | 12.58 | 27.80 | 45.03 | 31.73 | 86.62 | 24.68 | 22.00 |
| TODInv | P2P | Global | 9.48 | 28.59 | 34.90 | 26.83 | 87.40 | 25.89 | 21.62 |
| DDIM | MasaCtrl | Global | 25.61 | 23.45 | 85.26 | 70.79 | 82.75 | 23.15 | 21.10 |
| PNPInv | MasaCtrl | Global | 22.52 | 23.79 | 69.85 | 66.34 | 84.07 | 23.54 | 21.12 |
| TODInv | MasaCtrl | Global | 19.39 | 25.29 | 55.96 | 54.13 | 85.23 | 23.90 | 22.86 |
| DDIM | PNP | Global | 29.69 | 23.20 | 90.48 | 75.78 | 82.32 | 24.90 | 22.57 |
| PNPInv | PNP | Global | 27.09 | 23.38 | 84.53 | 73.56 | 82.56 | 24.81 | 22.51 |
| TODInv | PNP | Global | 26.74 | 25.17 | 70.53 | 51.64 | 84.47 | 25.45 | 22.06 |
| DDIM | P2P-Zero | Global | 57.89 | 21.92 | 125.83 | 112.53 | 79.43 | 23.16 | 21.10 |
| PNPInv | P2P-Zero | Global | 42.69 | 22.93 | 99.60 | 98.70 | 81.40 | 23.80 | 21.81 |
| TODInv | P2P-Zero | Global | 43.25 | 22.84 | 98.12 | 96.21 | 81.27 | 24.54 | 21.50 |

### A.4 More Qualitative Comparison with MasaCtrl, PNP, and P2P-Zero editing methods

The qualitative comparison based on MasaCtrl, PNP, and P2P-Zero editing methods are shown in Fig. 9, Fig. 11, and Fig. 10 respectively.

As shown in the $2_{nd}$ sample of Fig. 11, all competitors fail on local appearance editing. In the $3_{rd}$ sample, none of the competitors capture the editing instruction of “A black and white sketch”, and pay more attention to “Pink” incorrectly. The same problem also emerged on modifying the “Red drink” to “Red wine”. That evidences the effectiveness of our method in capturing semantic instruction.

As shown in the red cycles in $1_{st}$ sample of Fig. 9, most of competitors can not preserve the chains in the original image. Our TODInv is also skilled at object removal rainbow.

In Fig. 10, DDIM and PNPInv fail to preserve face details when editing the “Shirt” to “Sweater”, and they also failed to preserve the color of the bear. Our TODInv preserves more source details during editing. That should contribute to our task-oriented strategy, as we optimize the prompt embeddings that are irrelevant to the current editing, which preserves the source details effectively.

Figure: (a) Source
Refer to caption: extracted/5810451/figures/comparison/Source/000000000021.jpg

Figure: (a) Source
Refer to caption: extracted/5810451/figures/comparison/Source/722000000000.jpg

Figure: (a) Source
Refer to caption: extracted/5810451/figures/comparison/Source/000000000031.jpg

### A.5 The Algorithm of TODInv

The algorithm of our TODInv inversion and editing can be seen in Alg. 1.

Figure: Algorithm 1 Algorithm of TODInv.

### A.6 Editing Type Determination

As discussed in Sec. 5, the main limitation of ToDInv is we need to determine the editing type before inversion. It may be not easy for unprofessional users. However, it is easy to determine the edit type based on the source and target prompts using ChatGPT. We present the illustration of determining editing types with ChatGPT in Fig. 12, we can see that it is easy to determine the editing type with our given prompts.

Figure: Figure 12: Illustration of determining editing types with ChatGPT.
Refer to caption: x5.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See weakness
soundness: 2
strengths: The authors proposes layer-wise prompt optimization for inversion
summary: The paper proposes task-oriented prompt optimization for editing.
weaknesses: 1. The paper lacks novelty. The idea of layer-wise optimization comes from P+:Extented prompt optimization paper. Although P+ is extension from textual inversion, i think the proposed inversion is just simple modification of Null-text inversion with P+. 

2. The method lacks efficiency. It requires much time for optimization, which takes over 20 seconds. Since there are so many editing methods which enables editing within 5 seconds or even using single step, this proposed method has no clear advantage compared with those recent methods.

3. Although the method shows comparison between different inversion methods, the primary point for editing quality evaluation comes from subjective metrics. Please show some result on user study.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: -	Is there any evidence supporting the decision to class layer prompt embeddings into two groups based on resolution: the structural prompts in low-resolution layers and appearance prompts in high-resolution layers? It would be helpful to adding experiments to illustrate this.
-	How to define or optimize the fixed $p$ (in fig. 4) that not belong to structure and appearance prompt set?
-	Minor issues: typesetting problem in line 343; overlapping elements in fig 3.
soundness: 2
strengths: -	The inversion method seems to perform well on multiple editing models.
-	The method is clearly descripted.
summary: The authors proposed a framework that jointly optimizes inversion and editing processes within the extended $P^∗$ space, which is using distinct embeddings across different U-Net layers and time steps.
weaknesses: -	I have some concerns about the technical contribution of the paper. The main contribution claimed is the joint optimization process of inversion and editing framework. However, my understanding is that it only contains the inversion process that tries to optimize $P^*$. The second claimed contribution is the idea of classing different layer prompt embeddings into multiple groups according to the resolution. And this is adopted from a personalized editing work NeTI [1]. In general, I don’t think technical contributions of the paper meet the bar of ICLR.
-	The paper does not appear to achieve superior results compared to existing methods. The **qualitative** results (i.e., fig.1& 5) are similar to those from PNPInv, and SPDInv results are not so convincing to me. The results of SPDInv are largely unchanged in many cases (see fig.1& 5), which needs a reasonable explanation. Additionally, the paper lacks separate comparative evaluations for appearance, structure, and combined editing, as it claims task-oriented editing.

[1] Alaluf, Yuval, et al. "A neural space-time representation for text-to-image personalization." ACM Transactions on Graphics (TOG) 42.6 (2023): 1-10.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: please refer to the weaknesses.
soundness: 2
strengths: 1. The quantitative experiments are dense, which is good. 

2. The network structure figures are drawn in a professional way.

3. The method is explained clearly and the paper is easy to follow.
summary: TODInv is a finetuning required  text-guided image editing method, which categorized the image editing task into three subtasks, structure editing, appearance editing and global editing. For each class of editing, TODInv minimizes the approximation error by optimizing specific
prompt embeddings that are irrelevant to the current editing. TODInv  optimizes the prompt embedding in the previously well-studied extended p-star space for this three different tasks. Extensive quantitative experiments are conducted on PIE-bench, which demonstrated its effectiveness.
weaknesses: 1. The quality of the edited images are very poor compared with SoTA methods nowadays, considering that it is the end of 2024 now.  The techniques in this paper are still from almost two  years ago, which have been improved a lot in recent years. 

2. The method requires optimization. Yet its editing results are far worse than Imagic from CVPR 2022 and its later improved versions. The authors did not compare with Imagic and the later methods improving Imagic. Especially, Imagic and its improved versions are capable of conducting various non-rigid editing well, which TODInv should compare with in the TEdBench benchmark. 

3. The separated prompt embedding optimization according to different editing task is a waste of time considering the fact that there are many unified text-guided image editing methods already capable of conducting all image editing tasks well, including structure editing, appearance editing and global editing. 

4. This paper severely lacks novelty. The extended prompt embedding space has been well studied in 2023. The various disentangled properties of UNets have been thoroughly explored by many works in recent two years. The optimization techniques in this paper are disused nowadays.  


Overall, unfortunately, the quality and novelty of this paper is too far from ICLR standard.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - How are the editing types chosen on the qualitative experiments on PIE-Bench? Are they set manually or automatically labelled as described in A.6?
- What are the connections between the editing methods and the inversion methods according to the experiments? Should the inversion implementation (the number and positions of structure and appearance layers) change for different editing methods?
soundness: 2
strengths: - The idea is straightforward. 
- The proposed TODInv performs well on certain single object editing examples, editing flexibility and background preservation are balanced.
- By further categorizing the editing types into structure/appearance/structure & appearance, more fine grained qualitative and quantitative analysis are carried out on PIE-bench.
summary: The paper introduces Task-Oriented Diffusion Inversion (TODInv), a framework designed to enhance text-based editing capabilities for real images. TODInv operates by optimizing prompt embeddings within the extended P* space, utilizing distinct embeddings across various U-Net layers and time steps. The framework categorizes editing tasks into structure, appearance, and global edits, optimizing embeddings unaffected by the current editing task.
weaknesses: - The idea of extended textual inversion is not new (e.g. neti), the extension proposed in this paper seems less flexible and heuristic.
- The inversion and editing paradigm in Fig. 3 is not clearly explained by the caption. A dedicated paragraph to explain the process as well as its relation to previous editing pipeline should be added.
- The lacks evaluations in more complex scenarios such as multiple objects and larger area editing.
- Requires more manual setting which could lead to less stable performance in different scenarios.
- To demonstrate the robustness of the proposed method, as well as the soundness of the claims, additional ablation experiments could be helpful, such as:
    * changing the ranges/numbers of appearance and structure layers (to demonstrate the robustness)
    * changing the editing area scales for structure editing, (if the editing area is small, even though there is structure change, should it be appearance editing? If so the editing type should be categorized by editing area.)
    * using miss matched editing types, (to show the soundness of the proposed method) 
- Minor formatting issues, e.g., 
    * text overlapping in line 342.
    * Time (s) not Times (s) in Table 1

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Why does the method exhibit inferior performance at T=10 and T=100 compared to T=50? This observation appears counterintuitive, as diffusion models typically demonstrate improved performance with an increasing number of timesteps.
2. To substantiate the claim that ChatGPT can perform editing task classification, a quantitative analysis demonstrating its classification accuracy on the PIE-Bench dataset images using the ChatGPT API is necessary.
    * Given the high operational costs associated with the ChatGPT API, exploring the use of open-source MLLMs presents a viable alternative. Additionally, fine-tuning a smaller model on task-specific datasets for editing classification could potentially offer a more cost-effective solution while maintaining performance.
3. Can we use TODInv on DiT [1] style models too? Recent emergence of DiT architectures as alternatives to UNet raises questions about TODInv's potential applicability to DiT models. 
4. Why do the structural similarity results exhibit notably strong performance for P* while demonstrating significantly weaker performances for P, Pt, and P+ in Table 2?

[1] Scalable Diffusion Models with Transformers
soundness: 3
strengths: 1. The paper is well-motivated: Even within text-based image editing tasks, different editing prompts require distinct editing capabilities, making it challenging to faithfully execute all editing prompts with a single unified approach. This paper effectively identifies this limitation and proposes a new method based on this observation.
2. The method is simple and the paper is easy to follow.
3. The paper provides extensive quantitative and qualitative experimental results across various editing methods and inversion methods.
summary: This work points out that existing diffusion inversion methods achieve good reconstruction but limited editability due to their separate treatment of inversion and editing processes. To address this limitation, the authors propose a new diffusion inversion method, TODInv, for high-fidelity text-based real image editing.

The authors first categorize text-based image editing prompts into three classes: structure editing, appearance editing, and global editing. Inspired by the extended prompt embedding space P*, they observe that different layers in the U-Net architecture exhibit varying levels of importance depending on the editing class. Consequently, they perform diffusion inversion through prompt optimization in P* space only on the relevant layers for each editing class. They demonstrate the effectiveness of TODInv through extensive quantitative and qualitative experiments.
weaknesses: 1. Poor presentation
    * The paper contains several errors in highlighting the best values in bold within tables. Most of these errors occur when the proposed method's values are incorrectly marked in bold despite not being the best performing results. These errors could potentially mislead readers into overestimating the performance of the proposed method and significantly diminish the credibility of the reported experimental results.
    * There is a significant formatting error in [page 7, line 343] where text lines overlap. While such formatting issues might seem minor, given that the submission has reached the 10-page limit, resolving this overlap would likely cause the paper to exceed the page limit - a violation of ICLR's submission guidelines.
2. Limitations of proposed methods
    * The paper's core concept of "task-oriented" begins with classifying given editing prompts, but this classification relies on ChatGPT. However, the paper lacks quantitative analysis regarding ChatGPT's classification performance. Moreover, this dependency should be accounted for in the inference time measurements.
    * The method's requirement to pre-define class-specific editing layers within the U-Net architecture introduces architectural constraints that limit its broader generalizability.
3. The lack of implementation code limits reproducibility and credibility of the reported experimental results.