Fast Imagic: Solving Overfitting in Text-guided Image Editing via Disentangled UNet with Forgetting Mechanism and Unified Vision-Language Optimization

Authors: shiwen  zhang
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-22 19:41:48

Abstract: Text-guided image editing on real or synthetic images, given only the original image itself and the target text prompt as inputs, is a very general and challenging task. It requires an editing model to estimate by itself which part of the image should be edited, and then perform  either rigid or non-rigid editing while preserving the characteristics of original image.   Imagic, the previous SOTA solution to text-guided image editing, suffers from slow optimization speed, and is prone to overfitting since there is only one image given.  In this paper, we  design a novel text-guided image editing method, Fast Imagic. First, we propose a vision-language joint optimization framework for fast aligning text embedding and UNet with the given image, which is capable of understanding and reconstructing the original image in 30 seconds, much faster and much less overfitting than  previous SOTA Imagic.  Then we propose a novel vector projection mechanism in text embedding space of Diffusion Models,  capable of  decomposing the identity similarity and editing strength thus controlling them separately. Finally, we discovered a general disentanglement property of UNet in Diffusion Models, i.e., UNet encoder learns space and structure, UNet decoder learns appearance and texture. With such a property, we design the forgetting mechanism by merging original checkpoint and optimized checkpoint to successfully tackle the fatal and inevitable overfitting issues when fine-tuning Diffusion Models on one image, thus significantly boosting the editing capability of Diffusion Models. Our method, Fast Imagic, even built on the outdated Stable Diffusion, achieves new state-of-the-art results on the challenging text-guided image editing benchmark: TEdBench,  surpassing the previous SOTA methods such as Imagic with Imagen, in terms of both CLIP score and LPIPS score.

Main paper: 
 introduction: 
 Text-guided Image Editing (20) is a fundamental problem in computer vision, with a target text prompt indicating the editing intention to the given image. The approaches of text-guided image editing are generally categorized into optimization-based methods and non-optimization ones according to whether fine-tuning process is performed for reconstruction. Recent non-optimization editing methods (3; 31; 5; 33; 18; 1; 4; 32) are very efficient. Yet they either struggle on preserving the precise characteristics of original image during complex editing, or suffer from being incapable of performing sophisticated and accurate non-rigid edits. It is undeniable that fine-tuning a diffusion Figure 1 : Fast Imagic could be used for consistent and controllable keyframe generation for visual storytelling and movie generation, given one input image and target prompts. We list several samples with different random seeds for each target prompt. We demonstrate Fast Imagic is capable of controling multiple characters performing various actions at different scenes.Fast Imagic could also control each different character separately. Forgetting strategy on UNet's encoder with vector subtraction leads to high flexibility and success rate to change the spatial structures and actions, preserving appearance and identity by reserving UNet's decoder.In this paper, we are going to tackle the aforementioned issues of the SOTA optimization-based editing method, Imagic (14) . We name our text-guided image editing method Fast Imagic, which consists of two stages: fine-tuning and editing.For fine-tuning stage, with a generated source prompt from BLIP (17) to describe the original image, we design a vision and language joint optimization framework, which could be regarded as a variant of Imagic (14) by combining the first stage and the second stage of Imagic into one and using BLIP generated caption as source prompt instead of using target prompt as source prompt like what Imagic does. Such simple modifications are the keys to much faster convergence speed and less overfitting than Imagic. With our joint learning of image and source text embedding, the finetuning stage using one image with our Fast Imagic+Stable Diffusion 1.4 (26) takes 30 seconds on an A100 GPU, compared with 7 minutes with Imagic +Stable Diffusion (14) reported by Imagic paper. This leads to 14x speed up. BLIP generated source prompt also eases overfitting, which we will demonstrate in the ablation study. In addition, our joint vision-language optimization eliminates the strange random flip phenomenon reported in Imagic, i.e., the direction of objects in the editing results randomly flip, leading to editing failures. Our Fast Imagic always lock the correct direction of the objects the same as the original image, and only flips the directions of objects when the target prompt instructs so.For editing stage, we propose two novel methods, vector projection in text embedding space and forgetting strategy with a finding of a general UNet disentangled property. For the first time in the literature of text-guided image editing with Diffusion Models, we propose a novel vector projection mechanism in text embedding space of Diffusion Models, which is capable to separately control the identity and editing strength by decomposing the language representations into identity embedding and editing embedding. We explore the properties of vector projection and compare it with previous vector subtraction method utilized in Imagic to demonstrate its superiority on identity preservation. Finally, we discovered a general property of UNet structure in Diffusion Models, i.e., UNet encoder learns space and structure, UNet decoder learns appearance and identity. With such a property, we could easily tackle the fatal overfitting issues of optimization-based image editing methods in a very effective and efficient manner during sampling process instead of fine-tuning process, by designing a forgetting mechanism with model merging according to UNet disentanglement. Without intention to reveal authors' information, Fast Imagic has been completely open-sourced for more than one year (of course, the open-sourced project is not called 'Fast Imagic', thankfully not violating the double-blind rule). We were the first to discover and open-source the encoder-decoder disentanglement phenomenon in Diffusion UNet models more than one year ago, though such a property is re-discovered in some recent papers.To sum up, our main contributions are: 1. We present Fast Imagic, an efficient vision-language joint alignment framework, capable of performing both rigid and non-rigid text-guided image editing, while speeds up previous SOTA Imagic 14 times, completely solves the overfitting issue of Imagic. 2. We introduce a novel vector projection mechanism in text embedding space of Diffusion Models, which decomposes the target prompt representations into identity embedding and editing embedding. This improves Fast Imagic's capability for preserving more consistent characteristics of original image than existing methods. 3. We design a novel forgetting strategy via model merging based on our discovery on the disentangled UNet architecture of diffusion models, i.e., UNet encoder learns space and structure, UNet decoder learns appearance and texture. This allows us to effectively tackle the critical overfitting issue of optimization-based image editing methods, thus significantly boosting the editing capability of diffusion models.Our Fast Imagic achieves new state-of-the-art results on the challenging benchmark TEdBench (14) (even by using an outdated Stable Diffusion 1.4), surpassing previous SOTA Imagic built on Imagen in terms of both CLIP score (8) and LPIPS score (34) . 
related works: 
 Test-time fine-tuning image editing Diffusion Models have dominated text to image generation. DDPM (11) improves Diffusion process proposed by (29) on generating images. DDIM (30) accelerates the sampling procedure of Diffusion Models by making reverse process deterministic and using sub-sequence of time-steps. Dalle 2 (25) trains a diffusion prior to convert a text caption to CLIP (23) image embedding and then employs a Diffusion Decoder to transfer the generated CLIP image embedding to an image. Imagen ( 28) is a Cascaded Diffusion Model (12) , whose UNet is composed of three Diffusion Models generating images with increasing resolutions, employing the powerful T5 text encoder (24) for complex semantic understanding and generating sophisticated scenarios. Stable Diffusion (26) utilizes Variational AutoEncoders (16) to compress the training image to a compact latent space so that the UNets could be trained with low resolution latents in order to save computational resources.These models are pretrained on billions of data. For image editing task with one given image, DreamBooth (27) , textual inversion (6), Lora (13) , Imagic(14) etc., could be trained with one image and conduct the edit with text to image generation.Test-time fine-tuning free image editing There are some test-time finetuning-free methods, which do not require to optimize the diffusion model for each reference image. However, methods like SDEdit (18) ,DDIM inversion (30) , MasaCtrl (4), Elite (32) all struggle to preserve the characteristics during complex editing and some of them could also change the view, pose and background irrelevant to target prompt, which leads to editing failures. Other typical methods like PnP Diffusion (31) , Instruct Pix2pix (3), Prompt to Prompt (7) are incapable to conduct non-rigid editing and space-related editing.Drag Diffusion (19) , which extends DragGAN (21) , is only capable of performing space-related editing, which is just a portion of general image editing tasks. Instead, our Fast Imagic is a general text-guided image editing framework to conduct various kinds of image editing operations, including spatial transformations.3 Fast Imagic 
preliminaries: 
 Diffusion models (11; 29) consist of a forward process and a reverse process. The forward process starts from the given image x 0 , and then progressively add Gaussian Noise ϵ t ∼ N (0, 1) in each timestep t to get x t . In such a diffusion process, x t can be directly calculated at each timestep t ∈ {0, ..., T },x t = √ α t x 0 + √ 1 -α t ϵ t (1) with α t being diffusion schedule parameters with 0 = α T < α T -1 ... < α 1 < α 0 = 1 .In the reverse process, given x t and text embedding e, the time-conditional UNets ϵ θ (x t , t, e) of diffusion models predict random noise ϵ t added to x t-1 . With DDIM (30) , the reverse process can be computed as,EQUATIONWith Latent Diffusion Models ( 26), the original image x 0 is replaced by a latent representation z 0 obtained from a VAE (16) Encoder ε(x 0 ). The overall training loss is computed as,L = E zt,ϵt,t,e ||ϵ t -ϵ θ (z t , t, e)|| 2 2(3)In order to tackle such challenging text-guided image editing problems, we propose a image and text alignment framework via joint optimization of text embedding and UNet with the given image. Shown in Figure 2 , we introduce the overall design of our vision-language joint optimization framework.Source prompt generation. We first use BLIP (17) to generate a caption describing the original image, which is referred to as the source prompt. The source prompt is then fed to the text encoder of Stable Diffusion (26) , generating an embedding e src of source prompt. Previous three-stage editing method Imagic ( 14) regards target prompt text embedding as source one e src . We found that it is essential to use the BLIP caption instead of using the target prompt as a pseudo source prompt like Imagic. Otherwise such fine-tuning methods easily lead to overfitting issues, as demonstrated in the 5th column 'Imagic SD' of Figure 6 . This phenomenon indicates that using the BLIP caption as source prompt would result in better semantic alignment with the given original image than Imagic.Vision-language alignment with joint optimization. We choose to optimize UNet encoder blocks of 0, 1, 2 and decoder blocks of 1, 2, 3 in the UNet structure since we found that fine-tuning deepest features would lead to overfitting in our Fast Imagic framework, demonstrated in Figure 2 . Similar with Imagic, we regard source text embedding as parameters to optimize. Yet different with Imagic which optimizes text embedding and UNet in two separate stages, we found it vital to align the source text embedding and UNet parameters simultaneously, which is of great importance for faster convergence and better reconstruction quality than Imagic. In particular, due to a large domain gap between text and image, we use different learning rates for source text embedding (10 -3 ) and UNet (6 × 10 -5 ) with Adam Optimizer (15) . For faster training, since we only have a single training image, we repeat the tensors on batch dimension for batch-wise optimization with a batch size of 10. We use mean square error loss, and empirically found that stable reconstruction results can be achieved when the final loss is less than 0.03. With the batch size set to 10, the models are fine-tuned for 35 to 40 steps. We stop the training over 35 steps when the loss is less than 0.03, or stop at 40 steps at most. This fine-tuning process is significantly more efficient than Imagic, taking 30 seconds on a single A100 GPU. The training loss is computed as, L = E zt,ϵt,t,esrc ||ϵ t -ϵ θ,esrc (z t , t, e src )|| 2 2 (4) where the main difference with the training loss presented in 3 is that e src is considered as parameters to optimize. We first input the target prompt to the CLIP (23) text encoder of the Stable Diffusion model (26) , computing a target text embedding e tgt . With our learned source text embedding e src , we introduce two methods to combine e src and e tgt so that the merged text embedding can instruct the UNet to preserve characteristics of original image and also follow the target prompt. Given e src ∈ R B×N ×C and e tgt ∈ R B×N ×C , we conduct all vector operations on the C dimension to get the final text embedding e.Vector Subtraction. We use the same interpolation method as Imagic ( 14),EQUATIONAs shown in Figure 3 , the final text embedding e is obtained by travelling along vector subtraction e tgt -e src . In our experiments, we found that in most cases, γ goes beyond 1 when the editing is performed successfully. This leads to a problem that the distance between the final embedding e and the source embedding e src may be so far that the appearance of the edited object could change vastly.Vector Projection. We propose to use vector projection to better preserve the appearance of the original image. As shown in the Figure 3 , we decompose a target prompt text embedding e tgt into a vector along e src and a vector orthogonal to e src . We call the orthogonal vector e edit . We first calculate the ratio r of the projected vector on e src direction. 𝑒 !"# 𝑒 $%$ 𝑒 &'($ = 𝑒 $%$ -𝑟𝑒 !"# 𝑟𝑒 !"# 𝛼𝑒 &'($ + 𝛽𝑒 !"# (1 -𝛾)𝑒 !"# +𝛾𝑒 $%$ (1 -𝛾)𝑒 !"# +𝛾𝑒 $%$ 𝛼𝑒 &'($ + 𝛽𝑒 !"# 𝑟𝑒𝑐𝑜𝑛𝑡𝑠𝑡𝑟𝑢𝑐𝑡𝑒𝑑 𝑒 !"#Thus, we could get the e edit by computing e edit = e tgt -re src (7) To control the characteristics similarity and editing strength separately, we sum e src and e edit with two coefficient α and β, e = αe src + βe editEditing. We use DDIM sampling (30) with a classifier free guidance (10) to conduct the edit. The guidance scale is 7.5. For vector subtraction, we iterate over a range of γ ∈ [0. Forgetting mechanism Considering the fact that there is only one training image provided, in some cases the diffusion model still overfits, thus losing the editing capability, though joint vision-language alignment could ease the overfitting to some extent. The fine-tuning process is computational expensive compared to sampling process, thus we design a novel forgetting mechanism during sampling process to tackle the overfitting problem. The network is only fine-tuned once, and can be converted to multiple different networks during sampling process by merging certain fine-tuned parameters w learned and the corresponding parameters of original UNet (before fine-tuning) w orig , with a balance coefficient σ. In practice, we found that σ = 0 works in general, which means that we can simply replace the learned parameters with original parameters so that the network completely forgets these learned parameters. However, which paramters should be forgotten?EQUATIONDisentangled UNet Shown in Figure 4 , we found an general disentanglement property of UNet in diffusion models. The encoder of UNets learns space and structure information like the pose, action, position, angle and overall layout of the image, while the decoder learns appearance and textures instead. We were the first to draw a clear and universal conclusion on disentangled UNet and open-sourced our solution completely more than one year ago. In Figure 4 , given the target prompt and original image in the first row, we conduct Fast Imagic with forgetting UNet encoder, UNet decoder and nothing in the second, third and fourth rows respectively. We could see that without forgetting mechanism, four out of five cases are overfitting. By forgetting UNet encoder, the structure and space features are changed yet the appearance and texture are preserved. Vice versa for forgetting UNet decoder.Tackling overfitting with disentangled UNet and forgetting mechanism If the target prompt tends to edit space and structure information, for example, the pose or layout, we will choose to forget parameters of the encoder. If the target prompt aims to edit the appearance, the parameters of decoder should be forgotten. Currently we only apply the forgetting strategy when a text embedding e is obtained by vector subtraction in previous section. We will conduct a thorough exploration of forgetting mechanism with disentangled UNet in appendix due to page limit.The overall workflow of Fast Imagic is explained in Figure 5 . The fine-tuning stage is the same for all images. The diamonds in the figure indicate that the process depends on the users's choices and preferences. In practice, these user decisions can also be replaced by thresholds on metrics like CLIP  
experiments: 
 Benchmark. TEdBench ( 14) is one of the most difficult public-available text-guided image editing benchmarks. It contains 100 editings, with one target prompt and one image for each edit. These target prompts are very general with diversity, including but not limited to changing the appearance of objects, replacing certain parts of the image, changing the position, action and number of the object, editing multiple objects with complex interactions. In particular, non-rigid edits turn out to be very tough for many SOTA text-guided image editing methods. In terms of quantitative evaluation, we utilize CLIP Score (8) to measure semantic alignments with target prompt, and LPIPS score (34) and FID score (9) to indicate fidelity to the original image.joint vision-language alignment. We explore the importance of using generated caption as source prompt and joint alignment. We use BLIP generated source prompt to describe the original image, yet previous SOTA method Imagic uses target prompt as source prompt. Since target prompt indicates the editing target, it is obviously inconsistent with the original image. In Figure 11 , we show cases that do not need to use forgetting strategy from Figure 6 , so that we could remove the effects of forgetting strategy. If target prompt is used instead of BLIP generated source prompt, all these cases of Fast Imagic without using generated source prompt will overfit.We could also find it vital to align vision and language simultaneously instead of in separate stages in order to ease overfitting, by comparing 'Imagic SD' and 'Fast Imagic SD' columns in Figure 11 since all the cases in this figure do not use forgetting mechanism.decomposition in text embedding space. We compare two different reasoning methods to merge e src and e tgt to get the final text embedding e, shown in Figure 7 . These two methods are complementary to each other, with vector projection better at preserving the identity, and vector subtraction showing stronger editing capability. Thus in the workflow of Figure 5 , we use vector subtraction to be the default option. When the characteristics could not be preserved by vector subtraction, we switch to vector projection.We explore what effects could the location and amount of forgotten parameters cause to the editing results, in terms of UNet encoder in Figure 9 and UNet decoder in Figure 8 . The default strategies are 'encoderattn' and 'decoderattn' in Figure 5 . In fact, our findings on disentangled UNet could explain several phenomenons in controllable generation and image editing. 1. Why ControlNets (33) copy the UNet encoder branch instead of UNet decoder? Because UNet encoder controls the space and structure of generated image. ControlNet utilize extra condition to fix the layout and structure and does not control the texture. 2. Why methods like PnP diffuion (31) could not conduct non-rigid editing, for example, change the pose and action of objects? because they only operate on UNet decoder and fix the UNet encoder, which means that the space and structure cannot be edited.hyperparameters effects for text embedding interpolation We explore the effects of hyperparameters in vector subtraction and vector projection in Figure 10 and in Figure 13 .We compare qualitative editing results of our Fast Imagic with SOTA methods on several random test samples from TEdBench in Figure 6 , demonstrating stronger semantic alignments with target prompts and more precise identity preservation than other methods. Quantitatively, our Fast Imagic with the even outdated Stable Diffusion 1.4, surpasses the current SOTA Imagic+Imagen on TEdBench benchmark in terms of both CLIP Score, LPIPS Score and FID Score, shown in Table 1 . For FID score, we follow the advice of the authors by setting dimension to 192 since there are only 100 samples in TEdBench.Fast Imagic SD (ours) Target prompt a photo of a horse. DreamBooth UNet 
conclusion: 
 We present our Fast Imagic framework to tackle the challenging text-guided image editing problem. Fast Imagic speeds up previous SOTA Imagic by 14 times, and completely solves the overfitting problem of Diffusion Models when fine-tuning with only one image, via vision-language joint alignment and disentangled UNet with forgetting mechanism, and obtain new SOTA on TEdBench.  
decomposition in text embedding space: 
 We compare two different reasoning methods to merge e src and e tgt to get the final text embedding e, shown in Figure 7 . These two methods are complementary to each other, with vector projection better at preserving the identity, and vector subtraction showing stronger editing capability. Thus in the workflow of Figure 5 , we use vector subtraction to be the default option. When the characteristics could not be preserved by vector subtraction, we switch to vector projection.For the dog and the cat examples, vector projection can preserve more details of the appearance of the dog and the cat than vector subtraction. However, for a glass of milk and cookie example, vector subtraction performs better than vector projection which struggles to change the milk to juice and also introduces wave-like blurs in the image. We observe such phenomenons in many other cases for vector projection, which demonstrates that it is more suitable for edits where the identity of object should be kept instead of changed.The default workflow is in the highlighted flow in Figure 12 , where the default forgetting mechanisms are 'encoderattn' and 'decoderattn'. However, we still demonstrate in Figure 9 and Figure 8 how changing which part of parameters to merge the models influence the editing result. We first inference without forgetting strategies. If overfitting happens, we choose from the default 'encoderattn' or 'decoderattn' strategy according to the UNet property and target prompt intention. The 'encoderattn' means forgetting all encoder parameters except attention-related parameters. 'decoderattn' means forgetting all decoder parameters except attention-related parameters. The user may choose to forget more or fewer parameters according to the editing results, which we demonstrate and explain in 9 and 8.We explore the effect of hyper-parameters to the editing result in Figure 10 .The fine-tuning process of Imagic is composed of two stages, text embedding optimization for 500 steps and UNet optimization for 1000 steps. Our Fast Imagic employ unified vision language optimization with a batch-wise traing on 1 a100 GPU, which leads to 40 steps in total, speeding up Imagic for 14 times. This unified optimization of vision and lanugage goes beyond speeding up Imagic. It also eases the overfitting issue to some extent, shown in Figure 11 . The overall workflow is shown in Figure 12 . No matter what the input image is, we use the same set of hyper-paramters for finetuning stage. In the editing stage, the default workflow is to use vector subtraction with γ in the range of 0.8 to 1.6. In general, a proper editing result should already been obtained from one of these 8 images. However, if a perfect editing did not show up, there are two possibilities, overfitting or underfitting. Underfitting leads to the fact that the edited object suffers from identity shift, which means with the editing strengthened, the appearance of target object becomes gradually inconsistent with input image. In this circumstance, one needs to apply vector projection instead, which I will show in another paper with examples from TEdBench. The more often case is overfitting, which means that Fast Imagic could reconstruct the input image well yet cannot conduct the edit successfully. With the disentangled property of UNet, we could utilize the forgetting strategy to tackle the overfitting issue. If the target prompt aims to edit space and structure, one should use the default "encoderattn" forgetting strategy. If the target prompt aims to edit appearance For the first case where the input image is a polar bear on the ice field, the target prompt is "A polar bear raising its hand". To begin the workflow in Figure, we first run vector projection without forgetting strategy with with γ in the range of 0.8 to 1.6. Shown in Figure , we could find that we are facing the overfitting issue and the polar bear is incapable of raising its hands. Following Figure , we then run the default forgetting strategy on UNet's encoder, i.e. "encoderattn", which means that newly learned parameters of self attention blocks and cross attention blocks are preserved in UNet encoder and all learned parameters of UNet decoder are preserved as well. The hyper-parameter γ still ranges from 0.8 to 1.6. This time we could find successful edits in the results.First the effect of Fast Imagic is influenced by randomness. The fine-tuning process inevitably introduces randomness thus for some particular cases, we cannot guarantee to perfectly reconstruct the details of original image thus we have to run the fine-tuning stage several times for these challenging cases. The sampling procedure is also related to the initial random seed of reverse process, thus for some extremely challenging cases we have to sample tens of images or even hundreds, though rarely the case, before we could get a proper edited one.Second, the editing capability of Fast Imagic is restricted by the utilized Diffusion Model. If the target prompt cannot even be generated by the Diffusion Model itself, it is almost impossible to accomplish the edit according to the target prompt. For example, the prompt 'a sitting flamingo'  
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Comprehensively explained engineering solution to improve on the SoTA with interesting insights.
main_review: main_review: The work is focused on the setting of natural language based image editing programs which are fine-tuned on a given image to achieve granular control over how the image is edited. The paper provides insights into how the authors beat the SoTA for this particular setting i.e. Imagic, that deepen our understanding of how various components is such VLM systems behave.

There are multiple grammatical errors throughout the paper that detract from the high quality of the work itself. The appendix, which the reviewer is not required to read, is essential to the paper. Particularly the decomposition and disentanglement should have been further explored in the main paper as they are the most valuable insights in the reviewer's opinion.. Overall the paper feels like a compilation of insights organically gleamed from trying to engineer improvements on Imagic and less like an effort fundamentally understand or change the entire system.

Besides the above criticisms, the paper comprehensively explained everything. It clearly explained the setting of the problem, compared itself to multiple other works and justified its novelty and performance enhancements more than satisfactorily. It experimentally demonstrated  that it beat the SoTA and sufficiently proved its claimed contributions, which are impressive.
title: title: Review Fast Imagic.
main_review: main_review: **Summary**
The authors propose a variation of the existing Imagic framework for text-guided image editing. The main contributions are:
1. Simultaneously optimize the text embedding and the UNet rather than in two stages. 
2. Utilize BLIP captions as starting point for text embedding optimization rather than the target text
3. Explore alternative text embedding projection at inference time.
4. Design a UNet forgetting strategy to avoid overfitting in the Imagic framework

**Strengths and Weaknesses**

*Strengths*
1. The paper shows SOTA performance in the TEdBench
2. The paper is easy to follow and understand, it is well written and organized, with a few unclear details.
3. The paper explores novel ideas on the properties of denoising UNets in diffusion models, using BLIP for captioning and jointly optimizing text embedding and unet in Imagic.

*Weaknesses*
1. Despite showing SOTA performance, the TEdBench is a small benchmark of 100 image-text pairs.
2. Unlike Imagic, there is no user study backing the CLIP Score, LPIPS Score and FID Score. It is hard to tell if they are statistically significant as presented in the paper, given the size of the dataset. 
3. The interesting ideas presented are not backed by **quantitative** ablation studies and rely on anecdotal examples. All together it is hard to assess which ones of the contributions is actually impacting the performance.
4. It is unclear what configuration of the method is used to produce the SOTA metrics reported. 
5. Quantitative results only include Imagic and no other methods like SDEdit and others.
6. Even though it appears that the presented method is much faster than the original Imagic (30s vs 7m) it is unclear if the gains are driven by the adoption of BLIP embeddings or the joint optimization of embeddings and unet or both. Having an ablation study on this would be better. 

**Questions**
1. There is a claim in the original Imagic paper that getting a resulting embedding after optimization close to the target embedding is critical for a seamless interpolation between original an target embeddings:
*We run this process for relatively few steps, in order to remain close to the initial target text embedding, obtaining eopt. This proximity enables meaningful linear interpolation in the embedding space, which does not exhibit linear behavior for distant embeddings.*
Do you not observe this need at all since you propose utilizing BLIP instead?
2. What configuration was used to produce the SOTA results reported? In particular, what forgetting mechanism? What embedding interpolation method? Were they the same across samples or some version of the workflow in Figure 5 was used? And, if Fig 5 was used, was it automated? how?
3. The original Imagic paper presents vastly different training configuration (100 steps for embedding optimization and 1500 for unet optimization). Do you think this performance was optimized? Were you able to replicate the experiment with a smaller number of steps?
4. Given that the editing task is quite fine-grained and often involves small detail changes, CLIP Score and LPIPS seem weaker metrics than a user study. Can you provide a justification for the use of these metrics, which are not the central metrics reported previous SOTA work such as Imagic?

**Soundness**: 2

**Presentation**: 3

**Contribution**: 2

**Overall**: 6
title: title: Very good paper with minor flaws.
main_review: main_review: ### Summary 
In this paper, the authors presented a novel framework for image editing using Diffusion Models. In particular, the authors put emphasis on solving the problems with overfitting and high computation time. The presented method proved to be significantly superior to the baseline solution, both in terms of efficiency and inference time.

### Major strengths: 
* The authors performed an in-depth analysis of the problem and their method.
* The paper is well-structured and easy to read.
* The authors provided very interesting observations such as disentanglement in UNet or using BLIP captions to reduce overfitting. 
* The paper was designed with the usability of the presented method in mind, giving the reader many hints for how to utilize it effectively. 
* The experiments proved the effectiveness of the proposed method. 

### Major weaknesses:
* The authors did not perform quantitative analysis using reference methods other than Imagic + Imagen, despite performing the qualitative analysis using them. 

### Minor weaknesses: 
* In Figure 5, the arrows showing the flow are barely visible, the font varies in size and the whole thing is quite chaotic. In Figure 12, the bolding obscures some of the text. The whole thing looks quite amateurish and does not match the rest of the paper.
title: title: Fast Imagic offers a quick and effective approach to text-guided image editing, solving issues like overfitting and slow performance, but it could use more examples, extra benchmarks, and a deeper dive into how each new feature contributes.
main_review: main_review: Strenghts:
1. The 14x speed-up in optimisation compared to Imagic is an impressive result. Clever idea to merge the first two stages of Imagic into one and using BLIP-generated captions.
2. The authors convincingly argue that by separating the UNet encoder and decoder responsibilities (space/structure vs. appearance/texture), they can overcome the typical overfitting issues that occur with fine-tuning on a single image.
3. The vector projection technique introduced for separately controlling identity and editing strength in the text embedding space is an insightful addition. 

Weaknesses:
1. Limited qualitative examples. The paper presents only a few qualitative examples. Give the lack of examples in used dataset, extending this to show its application in diverse domains, such as fashion visual try-ons, could be possible solution to further highlight the method’s versatility.
2. Evaluation is performed on just one benchmark (TEdBench) and quantitative evaluation is done only against one method. Including more datasets or comparisons against a wider range of methods would provide a stronger validation of the approach’s generalization and robustness.
3. Limited quantitative ablation studies - while the paper highlights new techniques like joint optimization, vector projection, and the forgetting mechanism, it lacks a thorough quantitative ablation study to evaluate the individual contributions of these methods. This would provide clearer insight into how each component affects performance.
4. Although authors propose threshold approach, method requires some degree of manual effort. 
5. Minor typographical errors: “capablity” (line 178) and “seperately” (line 170).