LoMOE: Localized Multi-Object Editing via Multi-Diffusion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:48:09

Abstract: Recent developments in diffusion models have demonstrated an exceptional capacity to generate high-quality prompt-conditioned image edits. Nevertheless, previous approaches have primarily relied on textual prompts for image editing, which tend to be less effective when making precise edits to specific objects or fine-grained regions within a scene containing single/multiple objects. We introduce a novel framework for zero-shot localized multi-object editing through a multi-diffusion process to overcome this challenge. This framework empowers users to perform various operations on objects within an image, such as adding, replacing, or editing $\textbf{many}$ objects in a complex scene $\textbf{in one pass}$. Our approach leverages foreground masks and corresponding simple text prompts that exert localized influences on the target regions resulting in high-fidelity image editing. A combination of cross-attention and background preservation losses within the latent space ensures that the characteristics of the object being edited are preserved while simultaneously achieving a high-quality, seamless reconstruction of the background with fewer artifacts compared to the state-of-the-art (SOTA). We also curate and release a dataset dedicated to multi-object editing, named $\texttt{LoMOE}$-Bench. Our experiments against existing SOTA demonstrate the improved effectiveness of our approach in terms of both image editing quality, and inference speed.

Main paper: 
 introduction: 
 Diffusion models [39] [40] [41] have exhibited an outstanding ability to generate highly realistic images based on text prompts. However, text-based editing of multiple fine-grained objects precisely at given locations within an image is a challenging task. This challenge primarily stems from the inherent complexity of controlling diffusion models to specify the accurate spatial attributes of an image, such as the scale and occlusion during synthesis. Existing methods for textual image editing use a global prompt for editing images, making it difficult to edit in a specific region while leaving other regions unaffected [6, 32] . Thus, this is an important problem to tackle, as real-life images often have multiple subjects and it is desirable to edit each subject independent of other subjects and the background while still retaining coherence in the composition of the image. To this end, we propose Localized Multi-Object Editing (LoMOE).Our method draws inspiration from the recent literature on compositional generative models [3, 18, 25] . It inherits generality without requiring training, making it a zero-shot solution similar to [3] . We utilize a pre-trained StableDiffusion 2.0 [40] as our base generative model. Our approach involves the manipulation of the diffusion trajectory within specific regions of an image earmarked for editing. We employ prompts that exert a localized influence on these regions while simultaneously incorporating a global prompt to guide the overall image reconstruction process that ensures a coherent composition of foreground and background with minimal/imperceptible artifacts. To initiate our editing procedure, we employ the inversion of the original image as a starting point, as proposed in [37] . For achieving high-fidelity, human-like edits in our images, we employ two crucial steps: (a) cross-attention matching and (b) background preservation. These preserve the integrity of the edited image by guaranteeing that the edits are realistic and aligned with the original image. This, in turn, enhances the overall quality and perceptual authenticity of the final output. Additionally, we also curate a novel benchmark dataset, named LoMOE-Bench for multi-object editing. Our contributions in this paper are as follows:(1) We present a framework LoMOE, for zero-shot text-based localized multi-object editing based on Multi-diffusion [3] . Our framework facilitates multiple edits in a single iteration via enforcement of cross-attention and background preservation, resulting in high fidelity and coherent image generation. (2) We introduce a new benchmark dataset for evaluating the multi-object editing performance of existing frameworks, termed LoMOE-Bench. 
related work: 
 Image Synthesis and Textual Guidance: Text-to-image synthesis has made significant strides in recent years, with its early developments rooted in RNNs [31] and GANs [17] , which were effective in generating simple objects such as flowers, dogs and cats but struggled in generating complex scenes, especially with multiple objects [4] . These models have now been superseded by diffusionbased methods which produce photorealistic images, causing a paradigm shift [21, 40, 41] . In a separate line of work, CLIP [38] was introduced, which is a vision-language model trained on a dataset of 400 million image-text pairs using techniques such as contrastive training. The rich embedding space CLIP provides has enabled various multi-modal applications such as text-based imaged generation [12, 13, 16, 24, 36, 39, 40, 46] . Compositional Diffusion Model: Kim et al. [25] observe text-toimage models fail to adhere to the positional/layout prompting via text. Thus, compositional diffusion models try to address the task of image generation conditioned on masks, where each mask is associated with a text prompt. In Make-a-Scene [15] , the initial step involves predicting a segmentation mask based on the provided text. Subsequently, this generated mask is employed in conjunction with the text to produce the final predicted image. Methods such as ControlNet and GLIGEN [29, 49] have propose fine-tuning for synthesizing images given text descriptions and spatial controls based on adapters. Finally, methods like [3, 18, 25] , aim to utilise the pre-trained models and masked regions with independent prompts to generate images without re-training.Image Editing: Paint-by-Word [1] was one of the first approaches to tackle the challenge of zero-shot local text-guided image manipulation. But this method exclusively worked with generated images as input and it required a distinct generative model for each input domain. Later, Meng et al. [32] showed how the forward diffusion process allows image editing by finding a common starting point for the original and the editing image. This popularised inversion among image editing frameworks such as [24, 37] . This approach was further improved upon by adding a structure prior to the editing process using cross-attention matching [19, 37] . Moreover, there have been improvements in inversion techniques producing higher quality reconstruction which results in more faithful edits [23, 33] . However, many of the aforementioned methods generate the whole image from the inversion. This compromises the quality of reconstruction in regions where the image was not supposed to be edited. Recently [5, 8] try to address the problem with the above maskfree methods by incorporating an implicit masking strategy based on cross-attention masks similar to [11] . Thus reinforcing the notion that masking (either implicit or explicit) is essential for restricting the generation process to a certain region [2, 34] . However, when it comes to multi-object editing, these methods fall short on 3 counts: (1) editing multiple regions in one pass, (2) maintaining consistency between the edited and the non-edited regions of the image, (3) accumulating error over the multiple edit passes. Our method explicitly takes care of these aspects of image editing while incorporating all the advancements of our predecessor methods. 
proposed method: 
 Problem Statement: In a multi-object editing scenario, the objective is to simultaneously make local edits to several objects within an image. Formally, we are given a pretrained diffusion model Φ, an image x 0 from image space X ⊂ R 𝑤×ℎ×3 (x 0 ∈ X (for stable diffusion-based models, X ⊂ R 512×512×3 ), and 𝑁 binary masks {𝑀 1 , • • • , 𝑀 𝑁 } along with a corresponding set of prompts {𝑐 1 , • • • , 𝑐 𝑁 }, where 𝑐 𝑖 ∈ C, the space of encoded text prompts. They are used to obtain an edited image x * such that the editing process precisely manifests at the locations dictated by the masks, in accordance with the guidance provided by the prompts. Overview of LoMOE: Localized Multi-Object Image Editing (LoMOE) comprises of three key steps (a) Inversion of the original image x 0 to obtain the latent code 𝑥 𝑖𝑛𝑣 , which initiates the editing procedure and ensures a coherent and controlled edit (b) Applying the Mul-tiDiffusion process for localized multi-object editing to limit the edits to mask-specific regions, and (c) Attribute and Background Preservation via cross attention and latent background preservation to retain structural consistency with the original image. Fig. 2 depicts an overview of our method.In this work, we employ a pretrained Stable Diffusion [40] model, denoted as Φ. This model encodes an input imagex 0 ∈ R 512×512×3 into a latent code 𝑥 0 ∈ E ⊂ R 64×64×4 .Given an image x 0 and it's corresponding latent code 𝑥 0 , inversion entails finding a latent 𝑥 𝑖𝑛𝑣 which reconstructs 𝑥 0 upon sampling. We adopt a deterministic DDIM reverse process to model the inversion step [37] . This process is deterministic when𝜎 𝑡 = 0 ∀ 𝑡 ∈ [𝑇 ],where 𝜎 ∈ R 𝑇 + parameterizes the family Q of inference distributions [41] and 𝑇 is the number of timesteps. The latent 𝑥 𝑖𝑛𝑣 = 𝑥 𝑇 and the intermediate latents are related byEQUATIONwhere 𝛼 𝑡 represents a prefixed noise schedule and 𝜖 𝜃 (𝑥 𝑡 , 𝑡) is a neural network trained to predict the noise 𝜖 𝑡 added to a sample 𝑥 𝑡 . This network can also be conditioned on text, images, or embeddings [22] , denoted by 𝜖 𝜃 (𝑥 𝑡 , 𝑡, 𝑐, ⊘), where 𝑐 is the encoded condition and ⊘ is the null condition. In LoMOE, 𝜖 𝜃 is conditioned on 𝑐 0 , a text prompt encoded using CLIP [38] , during inversion.The underlying prompt is generated utilizing a text-embedding framework such as BLIP [28] on the image x 0 . Additionally, at each step during the inversion process, we softly enforce gaussianity using a pairwise regularization L 𝑝𝑎𝑖𝑟 [37] and a divergence loss L 𝐾𝐿 [26] weighted by 𝜆. This adaptation is inspired by findings in [37] , which highlighted deviations from the desired statistical properties of uncorrelated, white gaussian noise in the noise maps generated by 𝜖 𝜃 , leading to poor editability. Details of these losses can be found in Sec. 1 of the supplementary.The inversion step offers a solid foundation for the editing process, outperforming random latent initialization (Ref. Supplementary Sec. 2.1). However, employing a standard diffusion process for editing poses limitations in controlling local regions within the image via simple prompts. To address this challenge, we adopt a MultiDiffusion approach [3] for localized multi-object editing.For a diffusion model Φ, the backward process entails generating a sequence of latents {𝑥 𝑖 } 0 𝑖=𝑇 -1 starting from 𝑥 𝑇 , progressively denoising it over time. Here, 𝑥 𝑡 -1 = Φ(𝑥 𝑡 |𝑐), where 𝑐 is the encoded condition. Utilizing a deterministic DDIM reverse process,EQUATIONBy running this backward process with 𝑥 𝑇 = 𝑥 𝑖𝑛𝑣 and the source prompt 𝑐 0 , we obtain a reconstructed version, 𝑥 ′ 0 , of the original latent code 𝑥 0 . This step is termed the reconstruction phase. To address any deviations between 𝑥 ′ 0 and 𝑥 0 , we adopt a strategy of preserving noise latents during the inversion process [23] . Additionally, we store the latents 𝑥 ′ 𝑡 and cross-attention maps Ā𝑟 𝑡 (Sec.A simple approach to edit x 0 involves running a backward process with 𝑥 𝑇 = 𝑥 𝑖𝑛𝑣 and guiding it using a target prompt [32] . However, this method applies prompt guidance across the entire image, rendering the output susceptible to unintended edits. Thus, we propose a localized prompting solution, confining edits to a masked region. To edit 𝑁 regions corresponding to 𝑁 masks concurrently, one might initially consider utilizing 𝑁 + 1 distinct diffusion processes {Φ(𝑥𝑗 𝑡 |𝑐 𝑗 )} 𝑁 𝑗=0 .Here, {𝑥 𝑗 𝑡 , 𝑐 𝑗 } 𝑗 ≥1 denote the latent code and encoded prompt for mask 𝑗, while {𝑥 0 𝑡 , 𝑐 0 } correspond to those of the background (source image x 0 ). However, LoMOE adopts a single MultiDiffusion process [3] denoted by Ψ for zero-shot conditional editing of regions within all the masks.Given masks {𝑀 1 , • • • , 𝑀 𝑁 } and 𝑀 0 = 1 -𝑚𝑖𝑛{ 𝑁 𝑖=1 𝑀 𝑖 , 1}, with a corresponding set of encoded text prompts 𝑧 = (𝑐 0 , 𝑐 1 , • • • , 𝑐 𝑁 ), the goal is to come up with a mapping function Ψ : E × C 𝑁 +1 → E, solving the following optimization problem:EQUATIONStarting from 𝑦 𝑇 , Ψ generates a sequence of latents{𝑦 𝑖 } 0 𝑖=𝑇 -1during the backward process, where 𝑦 𝑡 -1 = Ψ(𝑦 𝑡 |𝑧). The objective in Eq. 3 is designed to follow the denoising steps of Φ as closely as possible, enforced using the constraint L 𝑚𝑑 defined as:EQUATION)where ⊗ is the Hadamard product. The optimization problem in Eq. 3 has a closed-form solution given by:EQUATIONThus, editing in LoMOE is accomplished by running a backward process using Ψ with 𝑥 0 𝑇 = 𝑥 1 𝑇 = • • • = 𝑥 𝑁 𝑇 = 𝑥 𝑖𝑛𝑣 and in turn 𝑦 𝑇 = 𝑥 𝑖𝑛𝑣 via a deterministic DDIM reverse process for Φ (i.e, Φ 𝑥 𝑖 𝑡 | 𝑐 𝑖 is given by Eq. 2). This step is termed the edit phase. Additionally, the latents and attention maps stored during the reconstruction phase are used to define losses (Sec. 3.3) that guide the edit. We observe that InstructP2P [6] tends to modify the whole image. GLIDE [35] often removes the subject of the edit in cases where it fails to generate the edit. DiffEdit [11] often fails to make a successful edit although it is based on Stable Diffusion. BLD [2] and SDInpaint [40] don't preserve the structure of the input and make unintended attribute edits to the masked subject. Finally, we observe that our proposed LoMOE makes the intended edit, preserves the unmasked region and avoids unintended attribute edits.Bootstrapping. To enable Ψ(𝑦 𝑡 |𝑐 𝑖 ) to focus on region 𝑀 𝑖 during the early stages of the backward process (up to timestep 𝑇 𝑏 , referred to as the bootstrap parameter), while incorporating the entire image context later on [3] , we introduce a time-dependency in 𝑦 𝑡 , as follows:EQUATIONwhere 𝑏 𝑡 serves as a background and is obtained by noising the encoded version of a random image with a constant color to the noise level of timestep 𝑡, i.e. 𝑏 𝑡 = 𝜉 (x) where x ∈ X and 𝜉 is the Stable Diffusion encoder. This contributes to improved fidelity in generated images, particularly in scenarios involving tight masks.While Ψ addresses multi-object editing, it faces challenges in (1) maintaining structural consistency with the source image and (2) faithfully reconstructing the background. To address these shortcomings, we introduce losses L 𝑥𝑎 and L 𝑏 as post-hoc guidances. These losses are jointly optimized at each iteration during the edit process, thereby constraining the diffusion process.Stable Diffusion [40] incorporate cross-attention (CA) layers [43] within 𝜖 𝜃 to effectively condition their generation on text. These layers facilitate interaction between image and text modalities during denoising, resulting in spatial attention maps for each textual token. These attention maps are represented as:EQUATIONwhere 𝑄 denotes the projection of intermediate spatial features from 𝜖 𝜃 onto a query matrix 𝑊 𝑄 , 𝐾 denotes the projection of the text embedding 𝑐 onto a key matrix 𝑊 𝐾 , 𝑑 signifies the latent projection dimension, and Ā𝑖,𝑗 represents the weight of the 𝑗 𝑡ℎ text token on the 𝑖 𝑡ℎ pixel. Studies [19, 42] validate that UNet encodings, especially CA maps, encode valuable information about structure and spatial layout. Consequently, constraints on intermediate CA maps can guide sampling and control generation, as shown in [14, 37] . While techniques such as mask-based blending [11] and attention injection [19] aid in preserving structure, they often yield suboptimal results (Ref. Table 1 ). Additionally, [19] suggests that attention injection may overly constrain geometry, favoring a softer constraint.In EQUATIONAdditionally, we incorporate a temperature parameter 𝜏 in Eq. 7 to ensure distributional smoothness (Ref. Supplementary Sec. 2.1).In order to preserve the background in the output, we match the backgrounds of the latents during the edit process (𝑦 * 𝑡 ) with those stored during reconstruction (𝑥 ′ 𝑡 ) at each timestep using a loss L 𝑏 .EQUATION)where 𝑀 0 is the background mask. This approach is preferred over simple copy-pasting of the background [11] to ensure natural and photorealistic edits while avoiding border artifacts through improved blending of multiple regions generated by separate diffusion processes.During each timestep of the edit process, we update the attention maps and latent vectors by optimizing the combined loss:EQUATIONEQUATION)where 𝜆 𝑥𝑎 and 𝜆 𝑏 represent the weights assigned to the crossattention and background preservation losses, respectively. The updated latent is given by: EQUATIONWe utilized Stable Diffusion v2.0 as our pretrained model Φ. Additionally, we set the hyperparameters: 𝜆 𝑏 = 1.75, 𝜆 𝑥𝑎 = 1.00, 𝜏 = 1.25, and 𝑇 𝑏 = 10, based on empirical validation conducted on a held-out set comprising ten images. The majority of our experiments were conducted on a system equipped with a GeForce RTX-3090 with 24 GB of memory. For multi-object edits involving more than five masks, we utilized an A6000 GPU with 48 GB of memory. The code will be made available post-acceptance. 
experimental setting: 
 We consider two sets of experiments: (a) single-object edits and (b) multi-object edits. For the multi-object editing experiments, while LoMOE can be employed as it is, we resort to iterative editing for other methods, specifically dealing with mask-based methods from Table 1 . We report both qualitative and quantitative outcomes of our experiments.For single-object edits, we utilized a modified subset of the PIE-Bench [23] dataset, supplemented with images from AFHQ [9] , COCO [30] , and Imagen [44] . For multi-object edits, we introduce a new dataset named LoMOE-Bench, comprising ∼1000 edit operations featuring images with 2 to 7 masks, each paired with corresponding text prompts. Each image has 4 masks on average yielding 15 edit combinations per image through combinatorial selection ( 4 resulting in ∼1,000 operations across diverse images. The details of the curated dataset can be found in Sec. 4.1 of the supplementary material. The LoMOE-Bench dataset will be made public in due time.𝑖=1 4 𝐶 𝑖 ),To obtain masks for LoMOE-Bench, we employ SAM [27] , where users can generate masks either by clicking on objects of interest or by drawing bounding boxes around them, as illustrated in Fig. 4 . Additionally, for masks required in addition tasks for both datasets, we developed a simple Python GUI where users can draw masks directly onto the target regions of the images.We benchmark LoMOE against SOTA, including SDEdit [32] , Instruct-Pix2Pix (I-P2P) [6] , MasaCtrl [7] , Null Text Inversion with Promptto-Prompt (NTI w/ P2P) [33] , GLIDE [34] , DiffEdit [11] , Stable Diffusion Inpaint (SDInpaint) [40] and Blended Latent Diffusion (BLD) [2] . Official implementations were used for all methods, except for SDEdit and DiffEdit. GLIDE, DiffEdit, SDInpaint, BLD, and LoMOE leverage masks, whereas the other methods operate on the whole image. Additionally, there are differences among the methods in terms of the types of text prompts they require. SDEdit, DiffEdit, NTI (w/P2P) and MasaCtrl necessitate both source and target text prompts, and I-P2P takes edit instructions as prompts, prompting us to extend PIE-Bench to accommodate these methods. Similar to LoMOE, GLIDE, SDInpaint, and BLD only use edit prompts corresponding to the masks. Finally, given the considerably noisy masks generated by DiffEdit, we opted to provide it with ground truth masks. We quantitatively analyze the edited images on a set of neural metrics, namely Clip Score (CS) [20] with both source and target prompts, Background (BG)-LPIPS [50] , and Structural Distance [10] . Additionally, we employed classical metrics such as BG-SSIM [45] .The neural metrics evaluate the perceptual similarity of the image, emphasizing realism. On the other hand, classical metrics focus on pixel-level similarity and doesn't comment on the realism or quality of the edit. In contrast to previous approaches, we propose evaluating edits based on the target CS and offer target prompts for all images in both datasets. This approach enhances the effectiveness of measuring edit quality, as a high target CS indicates successful editing. Finally, we also use state-of-the-art image aesthetic metrics such as Image Reward (IR) [48] and Human Preference Score (HPS) [47] which have not been used previously to evaluate editing, to the best of our knowledge. These metrics validate which method produces images that are pleasing to the human eye. To ensure robustness in our assessments, we averaged all the metrics over 5 seeds and reported the average standard error for all methods. Additionally, we conduct a subjective evaluation experiment to assess the quality of edits, described in Sec. 5.4. 
results and discussion 5.1 single object edits: 
 In comparing LoMOE with various baselines Table 1 , LoMOE demonstrates superior neural metrics, highlighting its proficiency in maintaining fidelity with source image and target prompt while making realistic edits. However, GLIDE outperforms LoMOE in classical BG-SSIM, suggesting a trade-off between realism and pixel-wise faithfulness, as observed in prior works [32] . While GLIDE does well on BG-SSIM due to its inpainting model design, it falls short on neural and aesthetic metrics, resulting in less realistic/incorrect edits. Other methods like MasaCtrl, NTI (w/P2P), and I-P2P perform well on target CS, but lack in other aspects, especially background metrics, due to their operation without a mask. Notably, instances where the target CS is close to the first-row in Table 1 suggest the absence of applied edits. Therefore, target CS is the most important metric in this context. Masked methods like DiffEdit, BLD, and SD-Inpaint collectively rank second best across most metrics, indicating the preference for utilizing a mask in our edit context. Qualitative evaluation in Fig. 3 provides visual comparisons. Finally, LoMOE achieves the highest scores in the aesthetic metrics indicating that it's edits have the least artifacts and are most pleasant to humans. Further, Fig. 6 validates this in the user study.Similar to our observations in single-object editing, LoMOE exhibits superior performance across all neural and aesthetic metrics in multi-object editing, except for source CS. This deviation is anticipated, given the substantial image transformations in multiobject editing. Ideally, such transformations lead to images that are markedly different from the source prompt and more aligned with the target prompt. Therefore, elevated BG-LPIPS and Structural Distance better indicate perceptual quality, while a high target CS signifies successful editing. Conversely, all other methods display a considerably lower target CS compared to source CS, indicating unsuccessful edits. Intuitively, as the number of edited objects increases, the source CS tends to decrease, while the target CS tends to increase. Furthermore, given our single-pass approach, we achieve significant savings in edit time compared to methods that perform multi-edits iteratively. Additional details can be found in Sec. 3.4 of the supplementary. Fig. 5 shows qualitative results on all the compared methods on a few sample images. This demonstrates LoMOE's impressive performance in preserving the intricate details during edits.LTo assess the significance of each loss component in LoMOE, we conducted a comprehensive ablation study, maintaining a fixed seed, 𝜏 and 𝑇 𝑏 . The findings presented in Table 3 reveal that incorporating L 𝑥𝑎 enhances neural metrics, contributing to the realism of the edited image. Meanwhile, the inclusion of L 𝑏 improves our classical metrics, enhancing the faithfulness of the edited image.Notably, these two aspects -realism and faithfulness are orthogonal qualities in image generation and editing. The combination of both losses in LoMOE yields improved performance, achieving a balanced enhancement in both the realism and faithfulness of the edit. Detailed ablation results for varying values of 𝜏 and 𝑇 𝑏 , can be found in Sec. 3 of the supplementary.We performed a user study using images from the single-object dataset to assess user preferences among images edited using the various baseline methods. We had 40 participants in the age range of 23-40. The majority of them expressed a preference for the edits generated by LoMOE over those from the other baseline methods.The results are summarized in Fig. 6 , and our observations from the user preference survey are as follows:The user study revealed that LoMOE is the most favored image editing method, with 46% of participants ranking it as their first preference and 37% as their second preference. Users expressed overall satisfaction with LoMOE's reliability, even in cases where edits weren't entirely successful. Following LoMOE, BLD and I-P2P garnered appreciation, with 25% and 13% respectively for first preference. However, BLD's failures were noted to be drastic, rendering some images unusable, while I-P2P's unintended background changes often resulted in visually appealing edits. GLIDE, DiffEdit, and SDEdit emerged as the least preferred methods, with only single-digit percentages for first preference. Dissatisfaction stemmed from GLIDE's tendency to replace subjects with poorquality targets, and users found DiffEdit and SDEdit to be similar, with the former preserving unmasked regions of input images. Overall, LoMOE stood out as the preferred choice, while BLD and I-P2P offered viable alternatives despite their drawbacks.  
limitations: 
 The limitations of LoMOE are illustrated in Fig. 7 . These limitations are inherent to its underlying architecture, which is shared by the broader class of models it belongs to. Although LoMOE utilizes stable diffusion for generation, there are instances where, despite generating a very high fidelity edit, the quote "monster in the woods" also appears on the body (Row 2, Col 1, Fig. 7 ) due to the model interpreting the prompt as a text generation task [2] . Additionally, although the model adheres to the prompt in adding clouds to the masked region (Row 1, Col 1, Fig. 7 ), the edit is not very realistic, which can be attributed to the realism and faithfulness trade-off, as discussed in Sec. 5.1. Furthermore, similar to other mask-based generation methods, our model faces constraints in generating beyond specified regions, such as changes in pose or scale, where the input and output silhouettes of the object in question differ. This limitation is evident in the mouse and unicorn edits (Col 2, Fig. 7 ), where the model is constrained by the mask and is, therefore, unable to create a smaller mouse inside the mask or the unicorn horn outside the mask. However, it is essential to recognize that this limitation prevents unintended edits, distinguishing mask-based editing models from mask-free editing frameworks. Despite these constraints, our model demonstrates effectiveness within its scope of capabilities while maintaining precision in complex edits. 
conclusion: 
 We present LoMOE, a framework designed to address a task of localized multi-object editing using diffusion models. Our approach enables (mask and prompt)-driven multi-object editing without the need for prior training, allowing diverse operations on complex scenes in a single pass, thereby having improved inference speed compared to iterative single-object editing methods. Our framework achieves high-quality reconstructions with minimal artifacts through cross-attention and background preservation losses. Further, we curate LoMOE-Bench, a benchmark dataset that provides a valuable platform for evaluating multi-object image editing frameworks. We believe that LoMOE would serve as an effective tool for artists and designers. 
 
 Appendix:  


Decision: This paper was reviewed by four experts, received two week accept, one borderline accept, and one borderline reject. The paper presents LoMOE for localized multi-object editing using diffusion models. LoMOE localizes multiple objects within an image by utilizing foreground masks and edits the masked content by text prompts. The pros including building a dataset LoMOE-Bench for the multi-object editing task, well-written, good performance, etc. 

The concerns raised by reviewers are mainly focus on method novelty, especially comparison with previous methods like MultiDiffusion. The AC checked the rebuttal and was convinced, but suggested the authors to include the discussion and add more comparison with object editing works like Object-level Image Editing via Referring Expressions in the final main paper. The authors are required to include the experiments and comparison with suggested references to the final version. A through proofreading/polishing is suggested to improve clarity and avoid typos.

Based on the reviews, AC found the proposed dataset LoMOE-Bench has potential benefit to the community, the AC recommend this paper for acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel localized editing method for generated images. The framework performs inversion as well as a multi-diffusion reconstruction process to preserve the background regions to be untouched.
strength_weakness: strength_weakness: - The proposed method outperforms existing methods.
- A novel benchmark is introduced to thoroughly evaluate localized image editing.
limitations: limitations: - Does the proposed method operate on real images? Are the images required to be t2i generated?
- What is the overall runtime of the proposed method?
- How are the images, prompts, etc. selected when constructing LoMOE-Bench? What are the criteria?
- Missing references on recent advances in multi-object editing [1][2].

[1] Tunanyan H, Xu D, Navasardyan S, et al. Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else[J]. arXiv preprint arXiv:2310.07419, 2023.
[2] Khandelwal A. Infusion: Inject and attention fusion for multi concept zero-shot text-based video editing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 3017-3026.
paper_summary: paper_summary: This paper proposes a diffusion model-based framework, LoMOE, for zero-shot localized multi-object editing tasks. The proposed framework consists of three key steps: inverting the input image to obtain the latent code; applying a MultiDiffusion process for localized multi-object editing; and preserving attributes and background using cross-attention maps and a background mask. For the multi-object editing task, they curated a new dataset, LoMOE-Bench, which contains 1000 edit operations featuring images with 2 to 7 masks, each paired with corresponding text prompts. The authors compare the proposed method with existing state-of-the-art methods and show the improved effectiveness of their approach in terms of both image editing quality and inference speed.
strength_weakness: strength_weakness: 1. The authors curate a dataset LoMOE-Bench for the multi-object editing task.
2. The paper is well-written and easy to follow.
limitations: limitations: 1. The contribution of this paper is incremental to MultiDiffusion. The zero-shot localized multi-object editing is mainly achieved by MultiDiffusion, and the preservation of structures is primarily achieved via cross-attention maps following Prompt-to-prompt. A thorough discussion of the differences between the proposed method and previous works would help reviewers better understand the contribution.
2. MultiDiffusion is not chosen as one of the baseline methods in the comparison experiments. The comparison experiments in Fig.3, Fig.5, Tab.1, and Tab.2 mainly show the advantage of MultiDiffusion over the other methods. 
3. As this method uses explicit masks during optimization, the proposed method might not work well when the desired target object has a different shape. Other comparison methods, such as Prompt-to-prompt, which do not use explicit masks, could handle these cases better.
4. In the first two rows in Fig. 5, the results obtained by the proposed method (last column) show the replaced objects appearing brighter and less well blended with the original background. How well this method can scale to more objects if the provided mask is larger than five? Also, the optimization time will scale when providing more masks. The optimization time is not well discussed in the implementation details.
paper_summary: paper_summary: This paper proposes LoMOE, a framework for localized multi-object editing using diffusion models. LoMOE enables precise and simultaneous modifications to multiple objects within an image by utilizing foreground masks and simple text prompts. It achieves high-fidelity image edits without prior training, making it a zero-shot solution. The framework is evaluated through both single-object and multi-object editing experiments, demonstrating its effectiveness. Additionally, this paper introduces a new benchmark dataset, LoMOE-Bench, dedicated to multi-object editing. This dataset facilitates the evaluation and comparison of different image editing frameworks.
strength_weakness: strength_weakness: 1. Compared to most methods that can only handle single-object editing, LoMOE can process multi-object editing in one go, improving editing efficiency.
2. As a zero-shot solution, LoMOE is ready to use without the need for prior training, which simplifies the usage process and lowers the application barrier.
3. The release of the new benchmark dataset LoMOE-Bench provides a standardized platform for the evaluation of multi-object image editing.
4. The popularity of LoMOE's editing results was assessed through user research, demonstrating its potential and user acceptance in practical applications.
limitations: limitations: 1.	The titles of the paper are inconsistent. The title in the submitted PDF is “LoMOE: Localized Multi-Object Editing via Diffusion,” while the title in the OpenReview submission system is “LoMOE: Localized Multi-Object Editing via Multi-Diffusion.”
2.	On line 193 of the paper, the number of parentheses does not match. There are two left parentheses and one right parenthesis.
3.	Equation 8 is confusing. The function $\epsilon_\theta()$ can only accept one $x_t$ and one $c_i$ as inputs at a time.
4.	In Equation 9, the dimensions of $\bar{A}_t^r$ and $\bar{A}_t^e$ might differ because $c_0$ and $c_i$ may contain different numbers of words.
5.	The illustration in the third step (multi-object edit) of the overall framework (Figure 2) is very confusing and seems inconsistent with the method described in Section 3.3.3. Figure 2 shows that $y_{t-1}^*$ is obtained through $x_t - \nabla$. However, Equations 11, 12, and 13 in Section 3.3.3 indicate that $y_{t-1}^*$ is obtained through $\Phi{ (x_t - \nabla) }$.
6.	The notation in the paper is confusing. In Figure 2 and Equation 10, $y_t^*$ should be represented as $y_{t-1}^*$ because it is derived from $\epsilon_\theta(x_t)$.
7.	The method section of the paper is not clearly written. There is a discrepancy between the overall framework and the text in the methods section.
paper_summary: paper_summary: This paper proposed a novel method for zero-shot localized multi-object editing, dubbed LoMOE. It utilizes foreground masks and corresponding simple text prompts that exert localized influences on the target regions resulting in high-fidelity image editing. For preserving the characteristics of the object being edited and the background, it includes a combination of cross-attention and background preservation losses. The authors also release a dataset for multi-object editing, named LoMOE-Bench. Multiple experiments are conducted to show that LoMOE achieves improvement in both image editing quality and inference speed.
strength_weakness: strength_weakness: 1.The writing is clearly and easy to follow. The figure 2 provides a detailed depiction of the entire process of the proposed method. The authors introduce cross-attention loss  and background preservation loss  for high-quality and background preservation, which are simple and reasonable, resulting in excellent editing results.
2.The paper provides a substantial number of experiments, which are extensive and comprehensive. The authors compare the method with SOTA in both single object editing and multi-object editing tasks. They also provide a dataset dedicated to multi-object editing, which is to provide new directions for future research.
limitations: limitations: 1. The cross-attention preservation loss updates each attention map of multi-diffusion process. However, previous works have shown the attention maps trend to overlap for similar objects, which could result in poor quality. Therefore, using masks in cross-attention loss may prevent such a problem. Why not use masks in the cross-attention preservation loss ?

2. The previous methods which optimize latent or attention map during the inference process usually employ the thresholds to prevent excessive interference. However, it seems that the joint optimization does not apply such thresholds. Hope the authors explain the reason for this.

3. There are still some experimental issues:
 - a.  In multi-object editing, the authors did not provide a qualitative comparison with other methods.
 - b.  Null text achieves good results in non-region editing, and it can also facilitate multi-object editing with text conditions. Hope the qualitative experiments comparing it with the proposed method can be added.
 - c. Hope to add a user study on multi-object editing.