Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach

Authors: qihe  pan
zhen  zhao
zicheng  wang
yiming  wu
wei  ji
haoran  liang
ronghua  liang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 09:28:43

Abstract: A plethora of text-guided image editing methods has recently been developed by leveraging the impressive capabilities of large-scale diffusion-based generative models especially Stable Diffusion. Despite the success of diffusion models in producing high-quality images, their application to small object generation has been limited due to difficulties in aligning cross-modal attention maps between text and these objects. Our approach offers a training-free method that significantly mitigates this alignment issue with local and global attention guidance , enhancing the model's ability to accurately render small objects in accordance with textual descriptions. We detail the methodology in our approach, emphasizing its divergence from traditional generation techniques and highlighting its advantages. What's more important is that we also provide~\textit{SOEBench} (Small Object Editing), a standardized benchmark for quantitatively evaluating text-based small object generation collected from \textit{MSCOCO}\cite{lin2014microsoft} and \textit{OpenImage}\cite{kuznetsova2020open}. Preliminary results demonstrate the effectiveness of our method, showing marked improvements in the fidelity and accuracy of small object generation compared to existing models. This advancement not only contributes to the field of AI and computer vision but also opens up new possibilities for applications in various industries where precise image generation is critical. We will release our dataset on our project page: \href{https://soebench.github.io/}

Main paper: 
 introduction: 
 The realm of text-to-image generation has witnessed tremendous advancements with the advent of recent diffusion models [12, 25, 27, 29, 34] , which have successfully revolutionized various tasks, including photo editing [15, 36] , and inpainting [2, 30] . These models have demonstrated remarkable capabilities in producing and manipulating salient objects within a picture, like the main subject of a picture, under the description guidance. The success of such models can be attributed to the effective cross-modal feature alignment between textual descriptions and the corresponding visual objects during the synthesis process. Such alignment facilitates a coherent and accurate translation of textual descriptions into visual representations, resulting in impressive outcomes in both image editing and in-painting tasks.However, a severe issue arises when the pending object is small. As shown in Fig. 2 (b), using small masks to guide the target object editing can lead to problems such as attribute leakage, poor quality, and missing entities. This issue is primarily due to the model's inability to focus on such a small region of interest in the description, which can result in the model's inability to generate objects that align well with the textual descriptions. For example, consider an image with a size of 512 × 512 and a small object whose bounding box typically occupies only 64 × 64 pixels. When performing multilevel cross-modal feature alignment, with U-Net as the backbone, the cross-attention map is progressively down-sampled to a small resolution of 8 × 8 as the network deepens. At the same time, our The input image with mask condition. The second column: Small mask condition is more likely leads to wrong attention map. The third column: Our method multi-scale joint attention guidance method can obtain the refined crossattention map for accurate small object editing. The last column: The output image generated by refined cross-attention map.target region may encompass within only a 1 × 1 grid, which is too small for the model to effectively focus on. Consequently, the model may struggle to generate objects that align well with the textual descriptions within such a small area, leading to poor quality and even missing entities.Facing the abovementioned limitations, we introduce a new task called Small Object Editing (SOE) in this work. Specifically, the SOE task requires the model to perform editing that is consistent with the given textual descriptions, seamlessly integrates with the surrounding context of the original image, and is precise in the desired small region. The key to this task is to prevent mismatches between the desired masked region and the intended textual description, ensuring that the model generates accurate and high-quality small objects. To evaluate the performance of models on this SOE task, we construct a comprehensive benchmark dataset, i.e., SOEBench. This benchmark allows us to evaluate the effectiveness of small object editing on various models and includes 4000 objects from two established datasets, MSCOCO [22] and OpenImages [18] . SOEBench holds two sets, SOE-2k and SOG-4k, where SOE-2k contains 2000 objects from OpenImages for editing and SOE-4k contains an additional 2000 objects from MSCOCO for editing. For each small object editing prompt, we provide both label-only and label-with-color templates in the description. This allows us to evaluate the models' ability to generate small objects that align with both textual descriptions and color specifications.Building upon the constructed SOEBench, we further provide a strong baseline method for small object editing. As discussed above, the quality of the cross-attention map is crucial in small object editing. To this end, we propose a new joint attention guidance method to enhance the accuracy of the cross-attention map alignment from both local and global perspectives. In particular, we first develop a local attention guidance strategy to enhance the foreground cross-attention map alignment and then introduce a global attention guidance strategy to enhance the background cross-attention map alignment. Our proposed baseline method is training-free but highly effective in addressing the SOE problem.The contributions of our work are summarized as below:• By recognizing the limitations of current research, we define a new task of small object editing and come up with a new benchmark dataset SOEBench to evaluate the model's ability on small object editing. • We introduce a novel training-free baseline approach to address the small object editing challenge by considering both global and local perspectives. • The proposed method is evaluated on the proposed SOEBench dataset and achieves the state-of-the-art results. 
diffusion models: 
 Denoising diffusion based models (DMs) [12, 25, 27, 29, 31] and score-matching-based model [32] [33] [34] have become the de facto standard for image generation, surpassing Generative Adversarial Networks (GANs) [9, 35, 38, 39, 41] in performance and training stability. Denoising Diffusion Probabilistic Models (DDPM) [12] and Noise Conditional Score Networks (NCSN) [34] are the pioneer to employ denoising neural networks to reverse a predefined Markovian noising process on raw images. Latent Diffusion Models (LDM) [29] intuitively disentangle image synthesis into two stages: semantic reconstruction through a denoising process in latent space, and perceptual reconstruction with VAE [17] , such a strategy significantly enhances visual fidelity. Currently, the success of DMs has greatly driven a series of image generation and editing tasks.Image editing has become a popular domain, enabling personalized and customized image generation. Text prompts are predominantly employed as input conditions due to their flexibility. These methods can generally be classified into two categories: training-based and training-free. Training-based methods typically involve fine-tuning the entire denoising model or adapters. For example, Diffusion-CLIP [16] and InstructPix2Pix [4] fine-tune the entire diffusion model using collected text-image datasets. In contrast, Asyrp [19] and GLIGEN [21] introduce adapters into publicly available diffusion models and only tune the adapter parameters for rapid adaptation. T2I-Adapter [24] and ControlNet [40] further extend control capabilities by introducing tailored adapter architectures such as zero convolution.For training-free methods, manipulating cross-attention maps and fine-tuning text embeddings are mainstream techniques. Promptto-prompt [10] edits images solely through text prompts, injecting attention maps of the original image throughout the diffusion process. Attend-and-Excite [5] introduces GSN to intervene in the diffusion process to enhance subject tokens for generating multiple objects. DDIM Inversion [31] presents a deterministic process to invert images to noises. Inspired by DDIM Inversion, Null-Text Inversion [23] pioneers fine-tuning null-text embeddings to reduce the distance between the sampling trajectory and the inversion trajectory. Prompt Tuning Inversion [8] follows a two-stage pipeline, first encoding the image into a learnable embedding and then sampling the edited image by interpolating the target embedding and the learnable embedding. PnP Inversion [14] enhances the inversion process by disentangling source and target branches for content preservation and edit fidelity, respectively. Imagic [15] incorporates text embedding optimization and model fine-tuning for image editing. BlendedDiffusion [1, 2] explore the method of using a mask to edit the specific region and add a new object to the image while leaving the rest unchanged. Some layout2image methods [6, 7, 26, 37] operated constraints on cross-attention to control the synthetic contents.Most text-based image editing methods are based on DMs, while a U-Net architecture is widely adopted. However, as the U-Net deepens, the cross-attention map becomes extremely small, thus some small objects may only take a small region on the attention map, leading to the model hardly focusing on such small objects for editing.Although image editing has been widely explored in recent years [3, [13] [14] [15] 36] , benchmarks for this task remain limited. EditBench [36] primarily presents a systematic benchmark comprising 240 images for image inpainting based on masks and text prompts, evaluated using CLIP-Score [11, 28] and CLIP-R-Precision metrics. Concurrently, Kawar et al. introduce TedBench [15] to evaluate non-rigid text-based image editing, providing 100 pairs of input images and target texts, with CLIP-Score and 1-LPIPS adopted as metrics. Edit-Val [3] introduces a standardized evaluation protocol for assessing multiple edit types using a curated image dataset. Additionally, Ju et al. present PIE-Bench [14] , comprising 700 images showcasing diverse scenarios and editing types. Aside from automatic evaluation via benchmarks, human evaluation (a.k.a. user study) is a more reliable method for assessing the quality of generated images. Text Alignment and Image Quality are two main metrics used in these studies. Text Alignment focuses on the consistency between the text prompt and the edited image, while Image Quality evaluates the visual fidelity of the generated image.However, to the best of our knowledge, there is no benchmark so far focusing on small object editing, which is a critical issue failed by most of the current research. Therefore, in this work, we focus on small object editing and propose a new small object editing benchmark, i.e., SOEBench. 
preliminaries 3.1 latent diffusion models: 
 Latent diffusion models are a subclass of generative models that adopt a diffusion and a denoising process to synthesize new data. First, a VAE encoder is adopted to encode the image I into the latent space, denoted as 𝑧. Then the forward diffusion process gradually introduces noise to the latent representation 𝑧 to the complete Gaussian noise 𝑧 𝑇 , where𝑇 represents the total number of timesteps. For each timestep t, the noisy latent code 𝑧 𝑡 can be represented as:EQUATIONwhere 𝛼 (𝑡) and 𝜎 (𝑡) control the mean and covariance of the noise.Then, the diffusion models aim to reverse this diffusion process by sampling random Gaussian noise 𝑧 𝑇 and gradually denoising such latent code to generate the initial latent code 𝑧 0 . In practice, the denoising model targets at predicting the sampled noise 𝜖 𝜃 at each timestep 𝑡 given the condition 𝑐 by optimizing the Mean Square Error between the predicted sampled noise 𝜖 𝜃 (𝑧 𝑡 , 𝑐, 𝑡) and the sampled noise 𝜖, which can be formulated as:EQUATIONFinally, a VAE decoder is adopted to generate the image using the denoised latent code.The Latent Stable Diffusion models perform conditional generation utilizing a cross-modal attention module between the given conditioning 𝑐 and the latent representations 𝑧 𝑡 , where the condition embedding 𝑐 and the latent representations 𝑧 𝑡 are mapped to the query Q with a dimension of 𝑀 ×𝑑 and the key K with a dimension of 𝑁 × 𝑑, and the cross-attention map A 𝑙 in the 𝑙-th layer of the U-Net can be derived as:EQUATION)Attention control is widely used in image editing, notably in techniques such as Prompt-to-Prompt(P2P). P2P replaces the current attention map with one corresponding to the target text, thereby facilitating the editing of objects within the image. 
the soebench dataset: 
 SOEBench is selectively extracted from MSCOCO [22] and Open-Image [18] databases, the selection rule of which follows a set of precise criteria to aptly serve the needs of small object generation experiments. The selection process primarily focuses on ensuring that the chosen objects are not obstructed by other elements within the image, allowing for a clear target for the small object generation task. Additionally, the object size was a critical consideration. We specifically selected objects that occupy a size smaller than 1/6 but larger than 1/8 of the overall image. This size constraint is pivotal because objects smaller than 1/8 would yield a representation on the deepest U-Net's feature map that is smaller than one pixel, rendering them practically unfeasible for effective generation, as shown in Fig. 4 .The category of the data contained in the dataset can be referred to in the word cloud in Fig 1 . Our dataset comprises about 300 types of objects that are frequently encountered, aligning the experimental setup with real-world object recognition scenarios. We then crop the images to extract the portions within the masked areas, and then use the BLIP-VQA [20] model to query, 'What is the primary color of the object in this area?' This approach efficiently identifies the main color of the objects within the masks as the color attribute. Importantly, the dataset has been segmented into two subsets based on the quantity of images they contain: SOE-2k and SOE-4k, where SOE-2k contains 2000 objects from the OpenImage dataset and SOE-4k contains 2000 more objects from the MSCOCO dataset. Such an operation enables a thorough and diverse assessment, ensuring that our methodology is tested under different scales of data availability, further reinforcing the validity and robustness of our research findings. Compared to existing benchmarks for text-based image editing (i.e., TeDBench [15] , EditBench [36] , and EditVal [3] ), our benchmark, SOEBench, offers distinct advantages. TeDBench is limited by its small dataset, consisting of only 100 images across 40 categories, primarily focused on modifying object attributes, states, or appearances. Furthermore, TeDBench images typically feature a single dominant subject occupying a significant portion of the image. In contrast, EditBench includes a larger dataset of 240 images and accommodates a wider range of mask sizes, from 0.08 to 0.9 of the image size. However, even EditBench's smallest mask sizes are substantially larger than those in SOEBench, where the maximum size of small object areas occupies only 1 36 of the image. Another recent benchmark, EditVal [3] , offers a more comprehensive set of image-editing operations. Developed from the MSCOCO [22] dataset, EditVal comprises 648 unique image-editing operations across 19 classes. These operations include 13 types of real-world edits, including one dedicated to controlling object size. Comparing SOEBench with these benchmarks (summarized in Tab. 1), we observe that SOEBench offers a larger and more diverse test dataset, covering a wider array of categories. Moreover, SOEBench features significantly smaller target areas, presenting a greater challenge for text-based image editing tasks. 
methodology: 
 In this section, we present our training-free multi-scale joint attention guidance baseline in detail. In particular, we first describe the problem definition of small objection editing in Sec. 5.1. Then, weU-net U-net + noise L lg L gg Z t Z' t m s m b + + Stage 1.A small cat on the grassK steps * J inters Z t (updated)Stage 2.-denoising UIn this paper, we propose a novel small object editing (SOE) task that targets at in-painting the specified masked region 𝑚 𝑠 on the given image 𝐼 , which is usually small, using a textual description 𝑐 as the condition. The task requires the model Ψ, which includes a VAE encoder E, a diffusion model with weight 𝜃 from which we can obtain the predicted noise 𝜖 and cross-attention map A, and a VAE decoder D, to not only perform inpainting consistent with the given condition but also perform the in-painting that integrates seamlessly with the surrounding context of the original image. Following previous works, a 𝐿-layer U-Net is adopted as the backbone of the diffusion model. The key to the task is to avoid any mismatch between the desired masked region and the intended textual description, which is usually achieved by refining the crossattention map A.In this paper, we aim at proposing a training-free baseline method to tackle the SOE problem. Based on the 𝐿-layer U-Net architecture, we focus on cross-attention map refinement and propose a local and global attention guidance method, where we encourage the refined cross-attention map to provide precise guidance to the latent code estimation. The overall inference pipeline is shown in Fig. 3 .During the inference stage, given the small mask 𝑚 𝑠 and the image 𝐼 , we implement local attention guidance according to the following steps. 1) we scale the height and the width of 𝑚 𝑠 according to the scaling factor 𝑠 to obtain a large mask 𝑚 𝑏 . Note the center position of 𝑚 𝑏 is identical to the center position of 𝑚 𝑠 . 2) Two sets of attention map A 𝑙 and A ′𝑙 can be obtained according to Eq. 4 and Eq. 5, whereEQUATIONandEQUATIONwhere F 𝜃 is the cross-attention map obtainment operation, where we collect the cross-attention maps from the diffusion model, 𝑧 ′ 𝑡 is the noised latent code predicted by using 𝑚 𝑏 as the input, which is initialized by 𝑧 ′ 𝑇 = 𝑧 𝑇 and 𝑙 denotes the layer index of the model. 3) We use the index of the label 𝑐 𝑟𝑒𝑞 in the given text condition 𝑐, e.g., we obtain the index of the label "cat" in the given text "A small cat on the grass", to index the attention maps desired, i.e., A 𝑙 𝑐 𝑟𝑒𝑞 and A ′𝑙 𝑐 𝑟𝑒𝑞 . 4) We crop A 𝑙 𝑐 𝑟𝑒𝑞 and A ′𝑙 𝑐 𝑟𝑒𝑞 according to 𝑚 𝑠 and 𝑚 𝑏 , respectively, to obtain the desired attention maps 𝑟 𝑚 𝑠 (A 𝑙 𝑐 𝑟𝑒𝑞 ) and 𝑟 𝑚 𝑏 (A ′𝑙 𝑐 𝑟𝑒𝑞 ). 5) We rescale 𝑟 𝑚 𝑏 (A ′𝑙 𝑐 𝑟𝑒𝑞 ) using bilinear interpolation to the same size as 𝑟 𝑚 𝑠 (A 𝑙 𝑐 𝑟𝑒𝑞 ) to calculate 𝐿 𝑙𝑔 loss according to Eq. 6.EQUATIONBased on our observation, the model is prone to generate accurate contents within such a large region. However, such an operation may fail to handle the seamless integration with the surrounding context of the original image problem. Therefore, we just It should be mentioned that Eq. 6 is designed solely to modify A 𝑙 while A ′𝑙 remains unchanged, thus obviating the need for a back-propagation operation.Our local cross-attention guidance approach is training-free but boosts the accuracy of the desired region in the cross-attention map, which shows its great potential in tackling the SOE problem. However, as shown in Fig. 5 , as the desired mask region may be too small, the pixels on the cross-attention map may be attracted to the wrong region, like the background. To tackle the issue, in this section, we further propose a global attention guidance operation to alleviate the foreground noise, which guarantees that when utilizing 𝑚 𝑠 to produce a cross-attention map, the map accurately zeroes in on pertinent information within the masked areas associated with the object being generated, effectively discounting data from areas inconsequential to the textual depiction of the object.In particular, based on A ′𝑙 introduced in Sec.5.2, we encourage A ′𝑙 to provide global guidance for A 𝑙 to alleviate the problem of cross-attention matching to the wrong region. Different from local attention guidance that only focuses on the cross-attention map on the desired mask region alignment, we encourage A 𝑙 to be aligned with A ′𝑙 for all queries, which can be formulated asEQUATION)Recall that 𝑖 is the index for the 𝑖-th condition token, 𝐼 represents the length of the sentence. Also note that A ′𝑙 is still unchanged and only the value of A 𝑙 needs to be modified, and no back-propagation operation is required.The overall guidance function is defined as:EQUATION)Then, we compute the gradient of 𝑧 𝑡 and update the predicted 𝑧 𝑡 as:EQUATIONwhere 𝜂 is the learning rate. Based on our experiments, we found that we only need to perform our joint attention guidance on the cross-attention map for the former 𝐾 denoising timesteps, where 𝑧 𝑡 undergoes a series of updates. Specifically, it is refined through a gradient descent process aimed at minimizing L 𝑡𝑜𝑡𝑎𝑙 , which occurs 𝐽 times. The step-by-step methodology for our method is delineated in Algorithm 1.Our joint attention guidance method performs cross-attention map alignment from a local perspective and a global perspective, which provides a comprehensive correction mechanism. This method aims to enhance the model's ability to accurately focus and generate small targets that are consistent with text descriptions, and assign lower attention values to non-target areas, thereby addressing the key challenge of using diffusion models to generate small targets. 
experiments 6.1 evaluation metrics.: 
 In the evaluation of our methodology, we employed two metrics: the CLIP-Score and the Fréchet Inception Distance (FID). The CLIP-Score was used to quantify the similarity between the generated objects within the target area and the corresponding text descriptions. This metric effectively assesses how well our model's output aligns with the textual prompts, particularly focusing on the small objects generated. The FID Score was utilized to gauge the quality of the generated images in relation to the original images. Higher CLIP-Score or lower FID, indicate better performance.It's important to note that the generated objects occupy a relatively small proportion of the original image, using the entire image for metric calculation could lead to misleading results. Therefore, we apply a targeted approach by cropping the original images during the evaluation phase. This cropping is designed to focus on the Algorithm 1: Our joint attention guidance method 1 Input: the textual prompt 𝑐, the region mask 𝑚 𝑠 , the extracted latent code 𝑧, the number of total timestep 𝑇 , the number of guiding timestep 𝐾, the number of backward times 𝐽 , the diffusion model 𝜖 𝜃 , the cross-attention map obtainment operation F 𝜃 , the learning rate 𝜂. 2 Output: the estimated latent code 𝑧 0 .3 Initialization: 𝑧 𝑇 ∼ N (0, 𝐼 ), 𝑧 ′ 𝑇 ← 𝑧 𝑇 4 Scale: 𝑚 𝑏 ← enlarge 𝑚 𝑠 . # Fix the center of the small box, then enlarge the length and width by a factor of 𝑠.5 for 𝑡 = 𝑇 , . . . ,𝑇 -𝐾 do 6 𝑧 ′ 𝑡 -1 = 𝜖 𝜃 (𝑧 ′ 𝑡 , 𝑡, 𝑐, 𝑚 𝑏 , 𝑧) , A ′𝑙 = F 𝜃 (𝑧 ′ 𝑡 , 𝑡, 𝑐, 𝑚 𝑏 , 𝑧) 7 for 𝑗 = 0, . . . , 𝐽 -1 do 8 𝑧 𝑡 -1 = 𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐, 𝑚 𝑠 , 𝑧) , A 𝑙 = F 𝜃 (𝑧 𝑡 , 𝑡, 𝑐, 𝑚 𝑠 , 𝑧) 9Get Features to Compute the 𝐿 𝑙𝑔 :10A 𝑙 𝑐 𝑟𝑒𝑞 = A 𝑙 [𝑐 𝑟𝑒𝑞 ] , A ′𝑙 𝑐 𝑟𝑒𝑞 = A ′𝑙 [𝑐 𝑟𝑒𝑞 ] 11 𝑟 𝑚 𝑠 (A 𝑙 𝑐 𝑟𝑒𝑞 ), 𝑟 𝑚 𝑏 (A 𝑙 𝑐 𝑟𝑒𝑞 ) #features within 𝑚 𝑠 , 𝑚 𝑏 12 Rescale: 13 𝐼 (𝑟 𝑚 𝑏 (A ′𝑙 𝑐 𝑟𝑒𝑞 )) ← resize 𝑟 𝑚 𝑏 (A ′𝑙 𝑐 𝑟𝑒𝑞 ) downscaling the 𝑟 𝑚 𝑏 (A ′𝑙𝑐 𝑟𝑒𝑞 ) using bilinear interpolation .14 area of interest, ensuring a more direct and meaningful comparison between the generated object and the original context. This method of evaluation ensures that our metrics accurately reflect the model's performance in generating small objects, providing a true assessment of its capability in this specific task.L 𝑡𝑜𝑡𝑎𝑙 = L 𝑙𝑔 (𝑟 𝑚 𝑠 (A 𝑙 𝑐 𝑟𝑒𝑞 ), 𝐼 (𝑟 𝑚 𝑏 (A ′𝑙 𝑐 𝑟𝑒𝑞 ))) + L 𝑔𝑔 (A 𝑙 , A ′𝑙 ) , ∇ 𝑧 𝑡 = 𝜕 𝑧 𝑡 L 𝑡𝑜𝑡𝑎𝑙 , 𝑧 𝑡 = 𝑧 𝑡 -𝜂 ∇ 𝑧 𝑡We use the stable diffusion v1-5 painting model as the baseline method and use DDIM as the noise scheduler. Considering the resize operation on the mask area, a too-large resize ratio will bring a large loss error. We set the scaling factor 𝑠 to a value randomly selected from 1.5 to 3. We set the learning rate 𝜂 = 100 by default. The noise correction is performed during the initial 𝐾 = 5 steps of the denoising process and repeated 𝐽 = 5 times at each step. More ablation study about the hyper-parameter can be found in Tab.XX, and the results indicates that our method is not sensitive to the hyper-parameters, highlighting the robustness of our approach.We evaluate our approach on our proposed benchmarks SOE-2k and SOE-4k with two kinds of prompt templates, i.e., label and color+label, e.g., "a dog" and "a brown dog", and quantitatively compute the CLIP-Score and FID to prove the effectiveness of our cross attention correction. The result is shown in Tab. 2 and our method demonstrated a great improvement compared to the baseline models, exceeding the baseline method stable diffusion by a large margin of nearly 1 point in terms of the FID score on the SOE-2k subset, and around 0.7 points in terms of the FID score on the SOE-4k dataset.It can be also inferred from the table that given a detailed condition, i.e., color with label, the editing performance of our method exceeds the performance of given a simpler condition, i.e., label only, by a large region of around 0.5 points in terms of the FID score on the SOE-2k subset, indicating the effectiveness of our method of understanding different conditions. It should also be mentioned that our method is training-free, and the experimental results verify the great potential of our method on tackling the small object editing problem.We further analyze the effectiveness of the detailed module designs of our joint attention guidance method, which includes a local attention guidance (lg) method and a global attention guidance (gg) method. It can be inferred from Tab. 2 that our local attention guidance can efficiently improve the models' in-painting performance on the small object editing task, mainly attributed to our newly proposed mask region scaling-and-rescaling operation, which efficiently enables the model to generate precise content corresponding to the textual description, and can also be integrated with the surrounding context of the original image successfully. Based on our local attention guidance method, our global attention guidance method can further bring performance improvement, as our global attention guidance method further reduces the probability of the model modifying the background part, ensuring the accuracy of the modified part, and at the same time ensuring the quality of background image generation.To complement our quantitative studies above, we present qualitative results in this section. We provide samples and qualitative comparison with Stable-Diffusion Inpainting model in Fig 6 . As show in the examples, when using the SD-I model for generating images of small objects, issues frequently arise such as low quality, mismatches between text and images, or the absence of generated objects. However, as shown in the third column of each image group in the figures, our method has been able to address these issues to a considerable extent, enhancing the performance of the base model. Besides the automated metrics, we also incorporated a user study to align more closely with intuitive human preference. In the study, we aim to assess both prompt adherence and the overall image. We employed our methods and SD-I(Stable-diffusion-Inpainting) as comparison models, keeping the seed fixed to generate 25 pairs, totaling 50 images. Our study involved 100 total participants. Participants were tasked with ranking images of small target objects generated these two methods. Fig. 7 present the user study. The most important results are:Our training-free methods can enhances the base model's ability to improve both the image quality of generated objects and their alignment with the associated text to a certain extent. Method comparison with P2P. In the realm of image content editing, prompt-to-prompt (P2P) methods have proven to be highly effective for various image manipulation tasks. Nevertheless, they exhibit notable limitations in generating small objects, primarily due to the mismatch between text and the corresponding crossattention map. To further evaluate this, we conducted tests using a P2P-based method, wherein we attempted to transfer the appropriate attention map from the large mask to the smaller one.Unfortunately, this approach proved ineffective for the generation of small objects because it did not correct the erroneous regions of the cross-attention map. This further corroborates the effectiveness of our method in addressing this specific issue. Failure Cases. While our method effectively mitigates performance degradation through attention guidance, its efficacy remains constrained by the limitations of the base model. As depicted in Fig. 8 , we showcase instances of object omission and subpar quality in our generated images. To bolster the small object editing capabilities further, fine-tuning the model on dedicated small object images presents a straightforward solution.orange orange brown dog brown teddy bear red carFigure 8 : Failure cases. Due to the performance of the base model, issues such as object missing and low will still be encountered even our joint attention guidance approach is adopted. 
conclusion: 
 In this work, our research introduces two significant contributions to the field of SOE (Small Object Editing) using diffusion models. We firstly developed a new benchmark dataset specifically designed to evaluate small object editing. This dataset addresses the unique challenges and requirements of small object imagery, providing a comprehensive and targeted platform for testing and comparing different models. Alongside this dataset, we propose a training-free approach represents a major advancement in solving the issue of generating small objects. By focusing on the alignment of cross attention maps between text and small objects, our method effectively bridges the gap that has historically hindered accurate and detailed generation of small objects in response to textual descriptions. This alignment enables the model to more precisely interpret and render the intricate details required for small object generation .This baseline serves as an initial reference point for future research, offering a simplified yet effective framework for subsequent models to build upon and refine. Another critical aspect to consider is that our method operates within the constraints of the existing capabilities of the diffusion model's backbone. Since we do not modify the parameters of the diffusion backbone, the quality and the fidelity of the generated images are inherently bound by the performance of the underlying model. Therefore, while our approach enhances the model's ability to generate small objects in alignment with text descriptions, the overall effectiveness is still dependent on the inherent capabilities of the diffusion model. 
 
 Appendix:  


Decision: This paper considers a method for text-guided image editing, specifically targeting small objects in the image which are often difficult to generate. Next to a new methodology a new benchmark dataset will be released. All reviewers agree that this is a paper that fits ACM Multimedia well. The reviews have gone through an interesting transition where one reviewer went from weak accept to borderline accept, where a borderline reject went to borderline accept. All in all we now have three borderline accepts. So, the paper can be accepted.

Reviews: review: paper_summary: paper_summary: The manuscript presents a significant contribution to the field of text-guided image editing, particularly focusing on the challenging task of small object editing. The authors propose a novel, training-free approach that leverages the powerful capabilities of large-scale diffusion-based generative models, specifically Stable Diffusion. They introduce a comprehensive benchmark dataset, SOEBench, designed to evaluate the performance of text-based small object editing models quantitatively.
strength_weakness: strength_weakness: 1. The proposed method enhances cross-modal attention alignment through local and global attention guidance, which significantly improves the model's ability to accurately render small objects based on textual descriptions.

2. The authors present extensive experimental results, including quantitative metrics and qualitative examples, to validate their approach. The improvement in performance over baseline models is clearly demonstrated.

3. The user preference study adds a practical dimension to the research, highlighting the real-world applicability and user satisfaction with the generated outputs.
limitations: limitations: 1. The description of the local and global attention guidance methods lacks clarity in some areas. Specifically, in Section 5.2 (Local Attention Guidance), the process of "scaling-and-rescaling" the target region is not thoroughly explained. It would be beneficial to include more detailed steps or a pseudo-code to help readers understand how the mask regions are upscaled and integrated into the cross-attention map.

2. The paper does not provide sufficient information on the choice of hyperparameters such as the scaling factors, the learning rate η, and the number of timesteps K and J in Algorithm 1. Including a discussion on how these hyperparameters were selected and their impact on the results would enhance reproducibility.

3. The method's reliance on the accuracy of the initial cross-attention map may limit its effectiveness if the initial attention map is significantly misaligned. The authors should discuss how robust their approach is to variations in the initial map quality.

4. The user preference study, while valuable, could be subject to biases. Details on the selection of participants, the interface used for comparison, and the instructions provided to the participants should be included to ensure the study's robustness and reproducibility.
paper_summary: paper_summary: This manuscripts offers a dataset for small object editing. In addition, a training-free approach is proposed to address small object editing.
strength_weakness: strength_weakness: The benchmark has rich amount of data whose mask size is specifically targeting the small size. The training-free approach to implement small object editing is simple and innovative.
limitations: limitations: 1. It would be better to offer the full-resolution image or zoom-in version of the small object to evaluate the quality of the small object. 
2. It would be better if the manuscript can add Adobe Photoshop's generative fill's results in user preference study.
paper_summary: paper_summary: This paper explores text-guided image editing methods, particularly focusing on small object generation, using large-scale diffusion-based generative models. The method offers a retraining-free solution to align cross-modal attention maps between text and small objects, thereby improving the accuracy of rendering objects according to textual descriptions. Additionally, this paper constructs the SOEBench dataset, which serves as a standardized benchmark for quantitatively evaluating text-based small object generation. The research problem addressed in the paper is highly practical, and the proposed method is closely tied to the research problem. However, the authors lack experimentation, especially in comparison with other methods.
strength_weakness: strength_weakness: 1) The research problem is well-defined and has strong practical value. The proposed method effectively addresses the issue of small target editing in text-to-image tasks.
2) A new image editing method for small targets is proposed, enabling content editing in images without the need for retraining.
3) A Benchmark Dataset for small target editing is constructed, which surpasses existing datasets in terms of data scale and category richness.
4) Comprehensive analysis and discussion of the proposed method are provided, although comparison with existing methods is lacking.
limitations: limitations: 1) Figure 1, serving as the abstract figure, only annotates some small target regions without conveying the research methodology and contributions of the paper.
2) Experimental results mainly focus on the proposed method without comparison with relevant methods. However, research on text-to-image editing is extensive, highlighting a significant drawback of this paper.
3) Section 6.7 discusses a comparison with P2P methods but relies solely on textual discourse, making it difficult to convince readers; incorporating visual experiments would better illustrate the method's advantages.
4) The reference format is inconsistent, for instance, using abbreviations for CVPR in [4] while providing the full name for the reference [2] (In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition), and similar issues are present in many references.
5) It is recommended to include discussions on time complexity in the method analysis in the experimental section for a more comprehensive discussion of method practicality.