Attentive Linguistic Tracking in Diffusion Models for Training-free Text-guided Image Editing

Authors: bingyan  liu
chengyu  wang
jun  huang
kui  jia
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 07:17:12

Abstract: Building on recent breakthroughs in diffusion-based text-to-image synthesis (TIS), training-free text-guided image editing (TIE) has become an indispensable aspect of modern image editing practices. It involves modifying the features in attention layers to alter objects or their attributes within images during the generation process. Yet, current image editing algorithms still present certain difficulties and challenges when it comes to editing multiple objects within an image. In this paper, we propose VICTORIA, a novel approach that enhances TIE by incorporating linguistic knowledge when manipulating attention maps during image generation. VICTORIA leverages components within self-attention layers to maintain spatial consistency between source and target images. Additionally, a novel loss function is designed to refine cross-attention maps, ensuring their alignment with linguistic constraints and enhancing the editing of multiple target entities. We also introduce a linguistic mask blending technique to improve the retention of information in areas exempt from modification. Experimental results across seven diverse datasets demonstrate that VICTORIA achieves substantial enhancements over state-of-the-art methods. This work highlights the critical role and effectiveness of linguistic analysis in boosting the performance of TIE.

Main paper: 
 introduction: 
 Text-to-Image Synthesis (TIS) has emerged as a groundbreaking field at the intersection of computer vision and natural language processing (NLP), offering the capability to generate visually compelling images from textual descriptions. Pioneering models such as Stable Diffusion [28] , DALL-E 2 [26] , Imagen [30] , and the more recent DALL-E 3 [21] , have demonstrated an exceptional ability to produce artistically coherent images. This development has attracted significant attention and research interest from both the academic community and industry [21, 35] .Training-free text-guided image editing (TIE) has emerged as a significant research area, avoiding the necessity for extensive training on large datasets. Instead, this approach harnesses pretrained Text-to-Image Synthesis (TIS) models or algorithms to edit images directly using textual prompts. Current training-free TIE techniques [1, 2, 5, 10, 16-18, 23, 33] excel in tasks such as image translation, style transformation, and modification of visual attributes, all the while preserving the structural and compositional integrity of the source images. Notably, Prompt-to-Prompt (P2P) [10] skillfully modifies specific regions of an image by substituting the cross-attention maps (CAMs) corresponding to the target edit words within the source prompt. Furthermore, Instruct-Pix2Pix [1] refines an instruction-based model by creating an image translation training dataset using P2P-generated images.Despite their advancements, current TIE algorithms are not without limitations. As illustrated in Figure 1 , the prevalent TIE methods encounter challenges when editing multiple objects within an image. The complexity of multi-object editing amplifies issues such as object loss (e.g., a missing apple), absence of object attributes (like spots), and incomplete representations of the background. Building on the analyses of earlier works [4, 5, 15, 18, 34] , these deficiencies Compared to previous methods such as Pix2pixzero [23] , Masactrl [2] , PnP [33] , P2P [10] , our method successfully corrects the editing errors of object loss, attribute absence, and incomplete backgrounds while preserving the spatial structure of the original image.in editing accuracy are predominantly due to the imprecision of CAMs in representing multiple objects.In this paper, we introduce VICTORIA2 , a novel approach that enhances training-free TIE by leveraging linguistic knowledge to address the issue of lost editing targets, such as objects, attributes, and backgrounds, especially in multi-target image editing scenarios. VICTORIA improves upon diffusion-based TIE by manipulating intermediate representations within attention layers to generate images, while taking linguistic insights into account. Rather than replacing CAMs, our method utilizes keys and queries in self-attention layers of the source image. This approach maintains spatial consistency between the source image and the edited target image. While this technique successfully achieves spatial consistency, addressing the inaccuracy of CAMs during image generation remains a challenge. To tackle this, VICTORIA carries out a syntactic analysis of the target editing text to discern word relationships. Utilizing these syntactic relations, we can guide the refinement of CAMs in the diffusion model, thereby improving the accuracy of edits across multiple targets without modifying the underlying TIS model. In the experiments, VICTORIA demonstrates superior performance, aligning edits more precisely on both new and established public benchmark datasets.In summary, the contributions of our paper are as follows:• We have developed a novel TIE algorithm that incorporates linguistics-based tracking. This algorithm is designed to be training-free, offering a plug-and-play solution that is compatible with current popular TIE methodologies.• For TIE tasks involving multiple objects, we leverage linguistic analysis to enhance the editing of individual objects, thereby improving the capability for multi-object editing.We also fill the gap in TIE by crafting a dataset specifically for multi-object editing tasks. • VICTORIA achieves the state-of-the-art (SOTA) performance across seven TIE benchmark datasets, underscoring the pivotal role and potency of linguistic analysis in enhancing the capabilities of TIE. 
related works 2.1 diffusion-based tis models: 
 In recent years, significant advancements have been achieved in text-driven image processing with Generative Adversarial Networks (GANs) [9, 14, 24] and vision-language models such as CLIP model [25] . While GANs have exhibited proficiency in handling in-domain data, they struggle with larger and more diverse datasets. Diffusion-based models like DALL-E 2 [26] , Imagen [30] , and Stable Diffusion [28] have been instrumental in propelling TIS forward by demonstrating exceptional prowess in synthesizing high-fidelity images. Despite their success, these models do not inherently possess TIE capabilities for the images they generate. Our work explores training-free TIE algorithms that build upon diffusion-based TIS models, thereby extending their editing functionality.TIE is a crucial task involving modifying an input image with requirements expressed by texts. In the literature, TIE approaches can be summarized into two categories: training-free and trainingbased. Training-free methods [2, 5, 10, 17, 18, 22, 23, 33] are designed to manipulate image generation during the denoising process. For instance, SDEdit [17] innovatively adds noise to a selected guide image to serve as the initial noise, yielding notable results. P2P [10] alters cross-attention maps to control the relationship between an image's spatial layout. Training-based methods craft new, ideal images by tweaking the model with domain-specific insights [7, 12, 15, 29] or by integrating supplementary guidance data [1, 19, 35] . Specifically, ControlNet [35] and T2I-Adapter [19] allow users to navigate the direction of image generation using input images by altering additional network modules. However, current TIE methods, either training-free or training-based, need more analysis of the relationships in CAMs. Our paper investigates the roles of entities and their modifiers of prompts in TIE through syntactic analysis. 
victoria: proposed approach: 
 Given an image, which can be a synthesis image generated from prompts or a real image, our objective is to guide the diffusion model to perform multi-object, multi-attribute editing based on the prompt. Our goal is to ensure that the edited image preserves the spatial structure of the source image while conforming to the target text description. In this task, maintaining spatial consistency between images is fundamental. Inspired by prior work [2, 10, 16, 33] , we accomplish this through control over the self-attention mechanism. Additionally, two other aspects require careful attention Source prompt: a yellow ripe apple and a modern jugUnet V Q K V Q K !!! z !,#$% z !,&#' Target image Source image Unet V Q K V Q K !!! !!! × 𝑇 Gaussian noise: z ( L B ETarget prompts: a pink glossy apple and a spotted jugSelf-Attention Control z ',#$% !!! × 𝑇 VQK VQK Self-Attention LayersCross-Attention Layers LBE Linguistic Mask Blending and Linguistic Linking Enhancement Positive CAMs Groupℒ !"# $ ℒ !"# % ℒ &'( 𝑈𝑝𝑑𝑎𝑡𝑒 z ',#$% z ',&#' z ',#$% (ii) Dependency parsing (iii) LBE (i) Overall pipeline ℒ '&)Target prompts to guarantee accurate image editing. First, unlike preceding methods [2, 10, 16, 33] that depend on the diffusion model's inherent cross-attention layer, our approach actively intervenes in the CAM generation process during the creation of the target edited image. This ensures a precise alignment between input text and target image at the cross-attention level. Second, it is crucial to preserve the regions of the image that do not require editing, which is achieved through modified masks informed by the cross-attention. An overview of VICTORIA is depicted in Figure 2 , which illustrates how VICTORIA is stratified into three distinct parts: input processing, output region restriction, and enhancement. Here, the "Self-Attention Control" module is dedicated to preserving the structural integrity of source images. The "Linguistic Linking Enhancement" part enriches the generative outcomes in the focal editing areas, making it particularly adept for multi-object editing scenarios. Lastly, the "Linguistic Mask Blending" component is designed to safeguard the information in regions that remain unaltered.A primary challenge in TIE is preserving the spatial layout between the source and target images. Contemporary editing methods, such as P2P [10] , typically involve the replacement of CAMs during the generation process. However, as identified in previous research [16] , replacing CAMs may inadvertently transfer an excess of information from the source image, potentially leading to undesirable outcomes. Moreover, P2P relies on textual consistency between source and target prompts, thus complicating its direct use for editing images that lack text prompts.Figure 3 illustrates the impact of replacing CAMs and SAMs on TIE, where a car edited using CAMs exhibits loss of structure and color fidelity. While editing with both CAMs and SAMs improves structural reconstruction, it falls short in the accurate translation of color. Conversely, selectively replacing SAMs at a low ratio produces images that align more closely with the target prompt. Additionally, Figure 4 presents the editing outcomes across different self-attention layers. The findings show that replacements across all layers result in a target image that mirrors the source. In contrast, limiting modifications to the middle layers significantly alters the spatial structure of the target image.Based on the above observations, we extract queries (𝑄 𝑠𝑟𝑐 ) and keys (𝐾 𝑠𝑟𝑐 ) from self-attention layers, setting us apart from previous image editing methods [2, 10, 16, 33] . The process is formulated as follows:𝑀 𝑑𝑠𝑡 = 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 ( 𝑄 𝑠𝑟𝑐 • 𝐾 𝑠𝑟𝑐 𝑇 √ 𝑑 ) (1)where 𝑀 𝑑𝑠𝑡 is the SAM of the target image during the editing process, 𝑑 is the dimension of keys and queries. The final output is conceptualized as a composite feature, which synergistically blends the structural characteristics of the source image with the feature attributes of the target image, denoted as: 𝑓 = 𝑀 𝑑𝑠𝑡 • 𝑉 𝑑𝑠𝑡 where 𝑉 𝑑𝑠𝑡 refers to the self-attention values of the target image. These elements are seamlessly integrated into the corresponding selfattention layers for generating the target image. Compared with directly substituting SAMs of fixed layers, the advantage of our method is that it can accommodate popular acceleration algorithms such as flash-attention [6] , which do not explicitly compute attention maps. After ensuring the spatial consistency between source and target images, it is important to also ensure the consistency between the target image and the text description (i.e., prompt). However, in contrast to previous methods ( [2, 10, 16, 33] ), which solely entrusted the task of integrating the text to the diffusion model's built-in crossattention layer, the inaccuracies introduced by the diffusion model's cross-attention layer may lead to the failure of integrating text and image, especially in the case of multi-object, multi-attribute editing (as shown in the introduction). To address these challenges, we introduce the Linguistic Linking Enhancement (LLE) technique aimed at improving TIE in such demanding scenarios by refining the editing process. The core concept involves a subtle optimization of the target CAMs during the editing process, rather than adjusting the parameters of the TIS model itself. This step is highly efficient and is executed during model inference, thereby maintaining our approach as training-free.We first extract modifier-head relations from the input prompt by dependency parsing [11] . We define 𝑆 as the collection of modifierhead pairs from the parsing result. The LLE technique is applied to all CAMs with respect to 𝑆. Furthermore, we denote by 𝑑 (𝑀 𝑚 , 𝑀 𝑛 ) a distance measure between the CAMs 𝑀 𝑚 and 𝑀 𝑛 corresponding to the modifier-head pair (𝑠 𝑚 , 𝑠 𝑛 ) ∈ 𝑆, where 𝑠 𝑚 and 𝑠 𝑛 represent the modifier and the headword, respectively. For instance, considering the prompt "a pink glossy apple and a spotted jug", the set 𝑆 contains three pairs: 1) 𝑠 𝑚 = "pink" and 𝑠 𝑛 = "apple", 2) 𝑠 𝑚 = "glossy" and 𝑠 𝑛 = "apple", and 3) 𝑠 𝑚 = "spotted" and 𝑠 𝑛 = "jug".In LLE, we initially construct a positive loss to minimize the CAM distance between modifiers and headwords, defined as:EQUATIONFor instance, this mechanism encourages the CAMs with respect to "pink" and "apple" to be brought closer. Additionally, we design a loss function that quantifies the disparity between the words in 𝑆 and the rest of the words present in the target prompt, promoting the dissociation of word pairs that are not syntactically connected.Let 𝑈 represent the collection of words in the target prompt not included in 𝑆, and 𝑀 𝑢 denotes the CAM corresponding to an unrelated word 𝑢 ∈ 𝑈 (e.g., "a", "and"). The negative loss is hence defined as follows:EQUATIONIn this context, we employ the symmetric Kullback-Leibler (KL) divergence as the distance measure to evaluate the proximity between two normalized CAMs. Furthermore, to encourage the CAMs of entity words (e.g., "apple", "jug") in modifier-head pairs to have a high activation value, thereby focusing the attention map more precisely on the corresponding object region, we define the attention loss as follows:EQUATIONThe overall loss function for our LLE task is then given by:EQUATIONOur optimization strategy draws parallels with the approach in [27] ,where the loss is predicated on pairs of all linguistically-related words for the task of image generation. In contrast, LLE focuses specifically on edited words and their linguistically-associated counterparts within the target prompt. It also aims to enhance the specificity of CAMs related to objects, particularly tailored for TIE.The final aspect of our framework addresses the preservation of non-editing areas. To combat contamination in non-target regions during the denoising process, we have adopted a strategy that exerts control over the background using masks, inspired by DAAM [32] .Our method dynamically generates editing masks by capitalizing on CAMs during the generation process of the target image. We denote 𝑆 𝑤 as the set including the editing word 𝑤, as well as its associated modifiers and headword. Taking the prompt "a photo of a colorful ruffed grouse" as an example, 𝑆 𝑤 with respect to the editing word "colorful" encompasses not only "colorful" itself but also "ruffed" and "grouse". We let 𝑊 represent the set of all editing words and define 𝑆 𝑊 = 𝑤 ∈𝑊 𝑆 𝑤 to represent the aggregation of words that are semantically tied to any word in 𝑊 . Moreover, we represent each CAM associated with a term 𝑠 𝑖 ∈ 𝑆 𝑊 as 𝑀 𝑖 . The linguistic blending mask is generated as follows:EQUATION)where 𝜏 is a threshold empirically chosen from the set {0.3, 0.4, 0.5}. Let 𝑧 src and 𝑧 dst be the latent codes of the source and target images, respectively, during the diffusion denoising process. The resultant latent codes for the generated image (denoted as 𝑧 dst * ) can be determined by the following equation:𝑧 dst * = (1 -𝑀𝑎𝑠𝑘) • 𝑧 src + 𝑀𝑎𝑠𝑘 • 𝑧 dst (7)This equation facilitates a balanced blending of the source and target images, taking into account linguistic knowledge. For a visual representation of this blending process, readers are directed to Figure 5 for a simple example. We summarize the VICTORIA algorithm in the form of pseudocodes. Let 𝐷𝑀 (𝑧 𝑡 , 𝑃, 𝑡) be the computation of step 𝑡 of the diffusion process, which outputs the latent code 𝑧 𝑡 -1 , 𝑄 𝑠𝑟𝑐 , 𝐾 𝑠𝑟𝑐 (from self-attention layers) and the CAM 𝑀 𝑐𝑟𝑜𝑠𝑠 (from cross-attention layers, omitted if not used). We denote by 𝐷𝑀 ( * ) the diffusion step where we override 𝑄 𝑑𝑠𝑡 , 𝐾 𝑑𝑠𝑡 with additional given 𝑄 𝑠𝑟𝑐 , 𝐾 𝑠𝑟𝑐 , but keep the values 𝑉 from the editing image. Formally, Algorithm 1 is tailored for editing synthetic images. Algorithm 2 is adept at managing scenarios encompassing real-world images without source prompts, skillfully transmuting source images into latent noise representations via DDIM-inversion [31] . This conversion process through DDIM-inversion effectively transforms real images into latent noises, which then act as initial latent codes for source images.Algorithm end if 12: end for 13: (𝐼 𝑟𝑒𝑠 , 𝐼 𝑑𝑠𝑡 ) ←VAE(𝑧 0,𝑠𝑟𝑐 , 𝑧 * 0,𝑑𝑠𝑡 ); 14: Return (𝐼 𝑟𝑒𝑠 , 𝐼 𝑑𝑠𝑡 ). 
experiments 4.1 datasets and experimental settings: 
 Considering the scarcity of public datasets for verifying the effectiveness of Text-Image Editing (TIE) algorithms in scenarios involving editing of multiple objects, we developed two types of evaluation datasets for assessing our approach in both synthetic and real-image scenarios, which will be released in public. The first, a synthetic image dataset named DVMP-edit-fake 3 , was specifically crafted to test the handling of editing adjectives and objects. It consists of 200 prompt pairs. For real-image scenarios, we introduced the DVMP-edit-real dataset, which comprises 808 imageprompt pairs. Moreover, for additional validation, we leveraged , PnP [33] , MasaCtrl [2] , SDEdit [17] w/ two noising levels, DiffEdit [5] , pix2pixzero [23] and InstructPix2Pix [1] . Only InstructPix2Pix is the trainingbased method.several publicly available image editing datasets created by in prior research [13, 16, 33] , including PIE-Benchmark [13] , Car-colorreal [16] , Car-color-fake [16] , ImageNet-R-TI2I [33] , and Wild-TI2Ireal [33] . Human evaluation, CLIP Score (CS), and Directional CLIP Similarity (CDS) [8, 25] are utilized to quantitatively analyze and compare our method against prevalent text-guided image editing (TIE) algorithms. The underlying TIS model employed in our experiments is the Stable Diffusion 1.5 4 . We generated experimental results for baselines using publicly available code from their original papers, ensuring consistency with unified random seeds.Figure 7 demonstrates the remarkable editing capabilities of our method across a range of scenarios. Our technique skillfully handles single attribute modifications, adeptly manages changes to attributes and backgrounds within generated scenes, and accurately replaces target entities. It also enables subtle shifts in visual style. Crucially, when confronted with complex editing tasks that involve multiple entities, our approach seamlessly executes simultaneous transformations on numerous entities and their attributes. Moreover, our method's independence from prompt alignment offers users the flexibility to precisely refine target images. This can be achieved through advanced prompt engineering techniques, including the utilization of prompt optimization tools [3, 20] .In this section, we first compare our method with SOTA methods [1, 2, 5, 10, 17, 23, 33] over DVMP-edit-real, DVMP-edit-fake datasets and PIE-Benchmark. Experimental results are presented in Figure 6 and Table 1 . As shown in Figure 6 , our method successfully converts real and synthetic images to target ones. In all examples, our method achieves high-fidelity editing that aligns with target prompts while preserving the structural information of source images to the maximum extent possible. Quantitative results [25] and CDS: Clip Directional Similarity [8, 25] .are presented in Table 1 . From Table 1 , our method outperforms all others in terms of CDS, indicating that our method excels in preserving the spatial structure and performing editing according to the requirements of target prompts, yielding superior results. We further carry out human evaluations over the Wild-TI2Ireal and ImageNet-R-TI2I datasets. In detail, from a pool of 228 examples, we invite a group of participants to choose what they consider the best TIE results by assessing source images alongside the target prompts provided. The findings are depicted in Figure 8 , illustrating the percentage of images the participants favor most. The figure showcases a comparison of user preferences for Ours and two strong baselines (i.e., P2P and MasaCtrl), as well as instances where the performance of the three methods appears to be on par. The result shows the superiority of our approach, with the highest rate at 43.64%.Linguistic Linking Enhancement. On DVMP-edit-real, we conducted ablation studies by incorporating different optimization losses proposed in Linguistic Linking Enhancement (LLE) under self-attention control , with results detailed in Table 2 and Figure 9 . Table 2 reveals that different optimization losses can improve the text-image consistency of edits while preserving spatial coherence, but with 𝐿 𝐿𝐿𝐸 performs the best. Figure 9 (a) demonstrates that when only using self-attention control (Self-Ctrl) for image editing, "a cartoon of a blue car" "a photo of a wooden car on the grass" "a sketch of a mountain" "a photo of a mountain above a lake " "a photo of a panda " "a painting of a tiger " "a flash apple and a furry monkey" "a flash tomato and a furry cat"From: To:Multi-Objective Real Image Editing both 𝐿 𝑒𝑛𝑡 and {𝐿 𝑝𝑜𝑠 , 𝐿 𝑛𝑒𝑔 } optimization result in missing object attributes or semantic misalignment. However, when using 𝐿 𝐿𝐿𝐸 loss, these errors are corrected. For example, in Figure 9 (a), 𝐿 𝐿𝐿𝐸 corrected the attributes of the "teal Striped cat"during the target editing. For single object edits, LLE enhanced object information (Figure 9 (c)) and corrected attribute alignment errors (Figure 9 (d)). For instance, in Figure 9 (d), both Self-Ctrl and Self-Ctrl + 𝐿 𝑒𝑛𝑡 wrongly aligned the attribute "yellow" to the "blanket, " which our method corrected. We further demonstrate the example of using LLE to correct the alignment of CAMs in Figure 10 . As demonstrated in Figure 10 , the leading cause of these failures is the text-image misalignment occurring in cross-attention layers. For instance, the word "strawberry" is incorrectly associated with the "candle" in the source image. The regions corresponding to the words "checkered" and "candle" are erroneously mapped onto "strawberry". Our method addresses these inaccuracies by employing LLE on CAMs to realign textual descriptions with appropriate image regions. This adjustment effectively corrects the editing errors, creating an edited image that accurately reflects textual descriptions. Linguistic Mask Blending. We execute ablation studies on Carcolor-fake and Car-color-real, with findings in Tables 3. During these studies, we benchmark the performance of P2P, which involves edits on CAMs against our approach, which operates on Self-Attention Control (Section 3.1). Additionally, we evaluate various blending strategies: no blending, blending focused solely on editing words (only editing words) and blending encompassing both editing words and their dependent counterparts (our approach). Our method outperformed P2P in terms of effectiveness, highlighting the advantages of our blending strategy, which proves beneficial for both P2P and our method. Conclusion also can draw from Figure 11 . As shown in Figure 11 , direct modifications to elements within self-attention layers can yield images that align with editing instructions but can alter contents in areas not intended for editing. To address this issue, we construct linguistic masks to preserve contents in non-editing areas. We successfully preserve source images' non-edited areas by employing this mask blending strategy. 
limitation: 
 The capabilities of the underlying TIS model may bind VICTORIA's performance. While VICTORIA can enhance the consistency of edited images, it encounters some limitations as the complexity of Table 3 : Ablation experiments of linguistic mask blending. Self-Ctrl is the method in Section 3.1.entities and the number of modifier words increase. The generation process sometimes fails when the underlying TIS model is unable to generate images aligned with the target prompt. VICTORIA is not good at removing an object by removing the corresponding word in the prompt. In actual image editing scenarios, the latent code derived from DDIM-inversion [31] carries priors of the source image, which can be unfavorable for regenerating images to fit certain prompts. Addressing these issues will be our future work. 
conclusion: 
 In conclusion, our proposed VICTORIA approach significantly advances the field of TIE. By leveraging linguistic insights, VICTORIA effectively addresses common challenges in existing TIE methods, especially in the case of multi-object, multi-attribute editing, such as maintaining spatial consistency and aligning attention maps with textual semantics. Our innovative linguistic mask blending technique and tailored loss function have demonstrated their effectiveness, exhibiting superior performance across multiple datasets. This work emphasizes the critical role of linguistic analysis in multiobject entity image editing and paves the way for new avenues of research and development. 
 
 Appendix: VICTORIA stands for attentiVe lInguistiC Tracking fOr tRaining-free text-guIded imAge editing. Source code and datasets will be made available upon paper acceptance.The DVMP-edit-fake dataset is a variant of the Diverse Visual Modifier Prompts (DVMP) dataset[27].App. & Style. Chg. Obj. & Style. Chg. 


Decision: The final ratings of this paper are two borderline rejects and two weak accept. The main concerns from the reviewers are novelty and experimental results. The rebuttal addressed some questions, the AC agrees with the reviewers that the paper has shown sufficient visual results and extensive experiments demonstrate the approach's ability. Therefore, the AC recommends accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper presents a text-to-image editing approach (VICTORIA) that leverages linguistic mask blending technique to achieve multi-object, multi-attribute editing tasks, and better preserve irrelevant content compared to prompt2prompt (P2P). VICTORIA consists three major modules: self-attention module, linguistic linking enhancement, and linguistic mask blending. Experiments are performed on various editing sets created by prior work[13, 16, 33].
strength_weakness: strength_weakness: - VICTORIA can preserve the irrelevant content of the image better compared to P2P. 
 
- Following P2P, VICTORIA operates in a training-free manner, which could be helpful in real-world applications.
limitations: limitations: - Although the paper mentions that a significant advantage of VICTORIA over previous methods is the preservation of text-irrelevant content, the proposed linguistic masking mechanism seems not that satisfying. For example, in columns 1, 2, 5, and 6 of Figure 1, the background parts should be preserved, yet the generated images still have different backgrounds. Considering the advancements in instance segmentation, many researchers currently use Grounded-SAM to preserve text-irrelevant regions. It is unclear why this paper did not adopt this approach or compare it.

- Another contribution of the paper is the ability of multi-object editing. On one hand, multi-object editing can be easily achieved by editing objects one-by-one. On the other hand, existing methods (*e.g.*, P2P, PnP) seem also have multi-object editing ability. In addition, when an image contains multiple objects, VICTORIA seems to randomly select an object to edit, and it is unclear how to precisely edit the target object (*e.g.*, Figure 6, "a purple ripe strawberry.").

- Table 1: Standard evaluation metrics such as DINO, FID, and GPT-4V are not reported.

- There is no discussion of the method's limitations. For example, VICTORIA does not seem to support non-rigid editing, like turning a sitting dog into a running dog as Imagic, Dreambooth did. The preservation of text-irrelevant content is also not very accurate.
paper_summary: paper_summary: The paper focusses on the task of text-guided image editing, where the authors have proposed a training-free method for editing images by utilising queries and keys from the self attention maps and also optimises the maps (guided by language parsing performed on prompt to capture relations)
strength_weakness: strength_weakness: well written and easy to follow paper; training-free approach; covers multiple scenarios of image editing like appearance, background, multi-object editing and so on
limitations: limitations: 1. It is unclear how the structural consistency part obtained via utilizing the keys and queries is different from the similar insight proposed in this paper on cross-image attention for appearance transfer [1]
2. Prompt2prompt's limitation on the requirement of knowing the prompt corresponding to the generated image is valid. But the authors should also try how their results compare will Null Text Inversion [2]
3. It is unclear why would the multi-object editing be a challenge with existing baselines?
4. Some results in the main paper (e.g., the "two birds" case in Figure 7 where the edit asks for "in front of the building) are not good/convincing
5. The proposed approach seems to be a combination of multiple existing baselines like [1] and [3] where [3] as well utilises clues from language parsing to optimize the attention maps. I agree that the authors maybe the first ones to show the applicability of such a method in the context of image editing, but more clarity into their novel technical contributions is required. 

[1] Cross-Image Attention for Zero-Shot Appearance Transfer
[2] Null-text inversion for editing real images using guided diffusion models
[3] Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
paper_summary: paper_summary: The paper introduces VICTORIA, a novel training-free text-guided image editing approach that incorporates linguistic knowledge to enhance the editing precision of multiple target objects within images by manipulating attention maps during the generation process.
strength_weakness: strength_weakness: VICTORIA addresses the challenge of multi-object editing by refining cross-attention maps through syntactic analysis of the target editing text, which is an innovative way to improve the alignment of textual descriptions with image content.
limitations: limitations: W1: As shown in Table 3, the improvement brought about by mask blending seems to be quite limited. However, it is the main contribution of this paper, questioning whether the enhancement all stems from Linguistic Linking Enhancement.

W2: The author lacks discussion on inference efficiency and how the computational cost of the method presented in this paper differs from previous methods.

W3: Formatting issue. The expression in this article has some inadequacies, as well as some issues with the formatting of symbols, for instance, the subscript of Loss should be written in upright type.
paper_summary: paper_summary: A novel approach is proposed in the paper to handle with TIE problem via self-attention control. In the framework proposed in the paper, multiple words amended in the editing prompt can correspond to different objects in the input image. More specifically, *Linguistic Linking Enhancement* is proposed to optimize the target CAMs instead of re-training or fine-tuning the diffusion models.
strength_weakness: strength_weakness: 1. Changes in multiple words in the prompt can be correspondingly reflected to the corresponding targets, breaking through the lack of relational understanding in LDMs under the control of cross-attention in the past and spatial information is simultaneously preserved. 

2. Sufficient visual results and extensive experiments demonstrate the ability of the approach. 

3. *Linguistic Mask Blending* enhances the ability of the framework to preserve the irrelevant regions, which is also remarkable. In addition, the editing results show that the abstract adjectives can be visualized in some ways, which has not been seen in previous works to the best of my knowledge.
limitations: limitations: 1. The author may show some of the failure cases in the paper. It can make the specific scenarios the model applies clearer to the reader, thus making it more attractive and objective.

2. The NLP toolkit in this framework can be replaced with LLM and the approach can be evaluated by MLLM for accuracy when editing for multiple objects, which can enhance the innovation and improve the persuasiveness of the paper.

However, I have the following questions about the ability of the approach: 

1. I am curious to know if *VICTORIA* can perform like approaches in papers like "Imagic: Text-Based Real Image Editing with Diffusion Models"[CVPR'2024] or "Edit One for All: Interactive Batch Image Editing"[CVPR'24]?

2. If I would like to remove an object from a picture, can I do it by removing the corresponding word in the prompt? (or replace the word with some special tokens?)