PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steering

Authors: yibin  wang
weizhong  zhang
jin  cheng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 04:54:51

Abstract: Image composition involves seamlessly integrating given objects into a specific visual context. Current training-free methods rely on composing attention weights from several samplers to guide the generator. However, since these weights are derived from disparate contexts, their combination leads to coherence confusion and loss of appearance information. These issues worsen with their excessive focus on background generation, even when unnecessary in this task. This not only impedes their swift implementation but also compromises foreground generation quality. Moreover, these methods introduce unwanted artifacts in the transition area. In this paper, we formulate image composition as a subject-based local editing task, solely focusing on foreground generation. At each step, the edited foreground is combined with the noisy background to maintain scene consistency. To address the remaining issues, we propose PrimeComposer, a faster training-free diffuser that composites the images by well-designed attention steering across different noise levels. This steering is predominantly achieved by our Correlation Diffuser, utilizing its self-attention layers at each step. Within these layers, the synthesized subject interacts with both the referenced object and background, capturing intricate details and coherent relationships. This prior information is encoded into the attention weights, which are then integrated into the self-attention layers of the generator to guide the synthesis process. Besides, we introduce a Region-constrained Cross-Attention to confine the impact of specific subject-related words to desired regions, addressing the unwanted artifacts shown in the prior method thereby further improving the coherence in the transition area. Our method exhibits the fastest inference efficiency and extensive experiments demonstrate our superiority both qualitatively and quantitatively. The code is available at https://github.com/CodeGoat24/PrimeComposer.

Main paper: 
 introduction: 
 Image composition entails seamlessly incorporating the given object into the specific visual context without altering the object's appearance while ensuring natural transitions. Earlier studies employ personalized concept learning [14, 15, 20, 21, 35] , yet they often rely on costly instance-based optimization and encounter limitations in generating concepts with specified backgrounds. Although these challenges are effectively addressed by utilizing diffusion models to explicitly incorporate additional guiding images [37, 46] , retraining these pre-trained models on customized datasets risks compromising their rich prior knowledge. Consequently, these methods exhibit limited compositional abilities beyond their training domain, and they still demand substantial computational resources.A recent study [28] develops a training-free framework, TF-ICON, that leverages attention weights from several samplers to composite the self-attention map of the diffuser during the composition process. Despite achieving notable success in this task, it still faces substantial challenges in preserving the appearance of complex objects (Fig. 2 (left)) and synthesizing a natural coherence (Fig. 2 (right)). The primary issue resides in its composite self-attention maps: the incorporation of attention weights from different contexts introduces potential ambiguity. To be precise, in TF-ICON, each sampler's weights are calculated within its specific global context [39] , and the act of forcibly combining them leads to the synthesized confusion in coherent relations and loss of appearance information. This undoubtedly hampers its ability to accurately represent the intrinsic characteristics of the given object and establish a coherent relationship. These issues are further accentuated by its overemphasis on background generation (a tailored sampler). It is essential to underscore that the background inherently necessitates no alteration, given the user's exclusive focus on foreground area generation. This superfluous emphasis not only introduces computational overhead but also leads to compromise on the foreground synthesis. Moreover, this approach introduces the unwanted artifacts in the transition area as shown in Fig. 4 , further hindering the natural coherence establishment.In summary, while the current training-free method has mitigated the need for costly optimization and retraining, it remains incapable of capturing the nuanced appearance of objects and forging dependable coherent relations. Urgent exploration of more effective steering mechanisms for training-free composition, without compromising efficiency, is imperative.In this paper, we novelly formulate this task as a subject-guided local editing problem, focusing solely on foreground generation due to the unnecessary alteration of the background. To achieve this, as depicted in Fig. 3 , we utilize the pre-trained Latent Diffusion Model (LDM) [34] to edit local foreground areas evolvingly based on the given object and text. Then the resultant edited area is spatially combined with a certain noised background at each step to maintain the scene. To address the remaining issues, we propose a faster training-free method, dubbed PrimeComposer, a progressively combined diffusion model that integrates the userprovided object to the background through well-designed attention steering across different noise levels. This progressive steering is primarily facilitated by our Correlation Diffuser (CD). Specifically, in each step, we combine specific noisy-level versions of the provided object and background at the pixel level while simultaneously segmenting the synthesized subject from the previous step's results. These components are then fed into the CD's diffusion pipeline where the synthesized subject interacts with both the referenced object and background within self-attention layers. The interrelation information, encoded as prior weights, encapsulates rich mutual correlations and object appearance features. Consequently, we infuse them into LDM's self-attention maps (yellow and orange regions in Fig. 3 (bottom right), respectively) to meticulously steer the preservation of object appearance and ensure harmonious coherence establishment. To fortify the steering impact, we further advance the classifier-free guidance [17] , elaborated in Sec. 4.5. Additionally, we introduce Region-constrained Cross-Attention (RCA), replacing cross-attention layers in LDM, to confine the impact of specific subject-related words to predefined regions in attention maps. This helps mitigate unwanted artifacts, thereby enhancing coherence in the transition area.Note that CD is the only sampler for steering and all the infused attention weights are computed from the accordant context. However, intuitively infusing object appearance-related weights on all layers will result in the subject overfitting, as shown in Fig. 5 , thereby leading to unexpected coherence problems: style inconsistency. Therefore, we propose to control appearance infusion only in the decoder part of the U-Net since the decoder has been proven to focus on learning the appearance and textures [52] .Our contributions can be summarized as follows. 1) We formulate image composition as a subject-guided local editing problem and propose a faster training-free method, namely, PrimeComposer. 2) We develop the CD to simultaneously alleviate the challenge of preserving complex objects' appearance and synthesizing natural coherence by well-designed attention steering. 3) We introduce RCA to address the unwanted artifacts in prior methods. 4) Our method exhibits the fastest inference efficiency and extensive experiments demonstrate our superiority both qualitatively and quantitatively. 
related work: 
 Image composition serves as a valuable tool for diverse downstream tasks, e.g., entertainment, and data augmentation [6, 11, 25, 28, 42] . The practice for this task broadly falls into two categories: textguided and image-guided composition.Text-guided composition [1, 2, 7, 12, 26] involves generating images based on a text prompt that specifies multiple objects. This approach allows for diverse appearances as long as the semantics align with the prompt. Despite its effectiveness, semantic errors may arise, especially with prompts involving multiple objects. These errors, including attribute leakage and missing objects, often necessitate extensive prompt engineering [43] . Conversely, image-guided composition [4, 13, 22, 28, 37, [45] [46] [47] incorporates specific objects and scenarios from user-provided photos, potentially with the assistance of a text prompt. This approach presents greater challenges, particularly when dealing with images from different visual domains. Specifically, image-guided composition encompasses various sub-tasks [31] , such as object placement [3, 8, 23, 38, 40, 41, 48] , image blending [44, 49] , image harmonization [9, 10, 19, 30, 32, 45] , and shadow generation [18, 24, 36, 51] . These diverse tasks are typically tackled by distinct models and pipelines, showcasing the intricacy of image-guided composition. Recently, diffusion models have demonstrated impressive capabilities in image-guided composition by simultaneously tackling all these subtasks. While prior studies have explored personalized concept learning [14, 15, 20, 21, 35] , they often rely on costly instancebased optimization and face limitations in generating concepts with specified backgrounds. To overcome these challenges, subsequent studies [37, 46] effectively incorporate additional guiding images into diffusion models through retraining pre-trained models on tailored datasets. However, this poses a risk of compromising their rich prior knowledge. Besides, these models exhibit limited compositional abilities beyond their training domain and demand substantial computational resources. A recent study [28] introduces a training-free method involving the gradual injection of composite self-attention maps through multiple samplers. Despite its remarkable success, it encounters challenges in preserving the appearance of complex objects and synthesizing natural coherence.Diverging from the approaches mentioned earlier, we novelly formulate this task as an subject-guided local editing problem. Our progressively combined Diffusion, PrimeComposer, intricately depicts the object and achieves harmonious coherence through welldesigned attention steering across a progression of noise levels. 
preliminary: 
 Denoising diffusion probabilistic models (DDPMs) [16] are designed to reverse a parameterized Markovian image noising process. They start with isotropic Gaussian noise samples and gradually transform them into samples from a training distribution by iteratively removing noise. Given a data distribution x 0 ∼ 𝑞(x 0 ), the forward noising process produces a sequence of latents x 1 , ..., x 𝑇 by adding Gaussian noise with variance 𝛽 𝑡 ∈ (0, 1) at each time step t:EQUATION𝑞(x 𝑡 |x 𝑡 -1 ) = N ( √︁ 1 -𝛽 𝑡 x 𝑡 -1 , 𝛽 𝑡 I).When 𝑇 is sufficiently large, the last latent x 𝑇 approximates an isotropic Gaussian distribution. An important property of the forward noising process is that any step x 𝑡 can be directly sampled from x 0 , without generating the intermediate steps:EQUATIONx 𝑡 = √ ᾱ𝑡 x 0 + √ 1 -ᾱ𝑡 𝝐,where 𝝐 ∼ N (0, I), 𝛼 𝑡 = 1 -𝛽 𝑡 , and ᾱ𝑡 = 𝑡 𝑠=0 𝛼 𝑠 . To draw a new sample from the distribution 𝑞(x 0 ), the Markovian process is reversed. Starting from a Gaussian noise sample x 𝑇 ∼ N (0, I) with 𝛼 𝑡 = 1 -𝛽 𝑡 , a reverse sequence is generated by sampling the posteriors 𝑞(x 𝑡 -1 |x 𝑡 , x 0 ). However, 𝑞(x 𝑡 -1 |x 𝑡 , x 0 ) is unknown and depends on the unknown data distribution 𝑞(x 0 ). To approximate this function, a deep neural network 𝑝 𝜃 is trained to predict the mean and covariance of x 𝑡 -1 given x 𝑡 as input:EQUATIONRather than inferring 𝜇 (x 𝑡 , 𝑡) directly, Ho et al. [16] propose to predict the noise 𝜖 𝜃 (x 𝑡 , 𝑡) that was added to x 0 to obtain x 𝑡 according to Equation 2. Then, 𝜇 (x 𝑡 , 𝑡) is derived using Bayes' theorem:EQUATIONFor more detail see [16] . In this work, we leverage the pre-trained text-to-image Latent Diffusion Model (LDM) [34] , a.k.a. Stable Diffusion, which applies the noising process in the latent space. 
method: 
 This section begins with an overview of our method, followed by an in-depth explanation of self-attention steering based on our Correlation Diffuser (CD). Subsequently, we will explore the details of our Region-constrained Cross-Attention (RCA). Finally, we will introduce our careful extension of classifier-free guidance (CFG) during inference.This work formulates image composition as a local object-guided editing task, utilizing LDM to depict the object and synthesize natural coherence. We aim to seamlessly synthesize the given object within specific foreground areas effectively without compromising efficiency. To achieve this, we propose PrimeComposer, a faster training-free progressively combined composer that composites the images by well-designed attention steering across different noise levels. Specifically, we leverage the prior attention weights from CD to steer the preservation of object appearance and the establishment of natural coherent relations. Its effectiveness is further enhanced through our extension of CFG. Besides, we introduce RCA to replace the cross-attention layers in LDM. RCA effectively restricts the influence of object-specific words to desired spatial regions, thereby mitigating unexpected artifacts around synthesized objects.In the depicted pipeline (Fig. 3 ), the process begins with a background image, an object image, a caption prompt 𝑃, and two binary masks M 𝑜𝑏 𝑗 and M 𝑓 𝑔 (designating the object and foreground areas, respectively). The background and object images are first inverted into latent representations z 𝑏𝑔 * 𝑇 and z 𝑜𝑏 𝑗 * 𝑇 using DPM-Solver++ [27] following [28] . These representations are then composited at the pixel level based on M 𝑜𝑏 𝑗 . To harness the prior knowledge of LDM for synthesizing the coherence, Gaussian noise is introduced to the transition areas (where M 𝑜𝑏 𝑗 XOR M 𝑓 𝑔 ), resulting in the initial input noise z 𝑖𝑛𝑖𝑡 . In each step t, we discern the attention weights embodying object appearance features and coherent correlations from CD's self-attention layers. These prior weights are then infused into LDM's self-attention maps to guide foreground generation. Additionally, all cross-attention maps in LDM are rectified to limit the impact of object-specific words in predefined regions. To preserve the unchanged scene, the edited foreground areas at each step are combined with the certain noisy version of the background z 𝑏𝑔 * 𝑡 -1 based on M 𝑓 𝑔 . This iterative process ensures seamless composition.While the composite noise z 𝑖𝑛𝑖𝑡 acts as the initial input, and the caption prompt 𝑃 contributes to inpainting the transition areas, LDM still encounters challenges in preserving the appearance of the object and synthesizing harmonious results effectively as shown in Fig. 7 . To tackle this, we propose the CD to chase down the attention weights that encapsulate rich prior semantic information of the object's features and coherent relations. Subsequently, these prior attention weights are employed to guide the synthesis process in the initial 𝛼𝑇 steps where 𝛼 is the hyperparameter.The CD is adapted from the pre-trained Stable Diffusion with tailored self-attention layers to generate prior attention maps. Specifically, at each timestep t, it takes the pixel composite image z 𝑝𝑐 * 𝑡 (derived from the specific noise version of the user-provided object and background) and the latent representation of the synthesized object z 𝑡,𝑜𝑏 𝑗 (segmented from the previous step's result z 𝑡 ) as input. In each self-attention layer l, the self-attention map A 𝑙,𝑡 are computed as follows:EQUATIONEQUATIONwhere reflects the relations between the synthesized object and background, while A 𝑜𝑏 𝑗 𝑙,𝑡 contains the object appearance features. These constituents are then infused into the l-th self-attention maps A 𝑙𝑑𝑚 𝑙,𝑡 in LDM, as illustrated by the yellow and orange regions in Fig. 3 (bottom right), respectively. The process can be formulated as: A * 𝑙,𝑡 = 𝜗 infusion (A 𝑐𝑟𝑜𝑠𝑠 𝑙,𝑡 , A 𝑜𝑏 𝑗 𝑙,𝑡 , A 𝑙𝑑𝑚 𝑙,𝑡 ), where 𝜗 infusion is the function to produce the infused self-attentionmaps A * 𝑙,𝑡 . However, the intuitive infusion of A 𝑜𝑏 𝑗 𝑙,𝑡 on all layers may result in the synthesized object closely resembling the given image, i.e., subject overfitting. This, in turn, can lead to unexpected coherence problems, as depicted in Fig. 5 . To address this issue, we propose a controlled approach to appearance infusion: restricting it only to the decoder part of the U-Net. This decision is grounded in the understanding that the decoder primarily focuses on learning appearance and texture [52] , thus promoting more natural coherence in the synthesized output.A 𝑙,𝑡 ∈ R (ℎ×𝑤 ) ×𝑛 , h,Following [28] , the caption prompt 𝑃 is utilized to guide the synthesis of transition areas. However, this introduces a coherence problem, as demonstrated in Fig. 4 and Fig. 7 : the newly generated object isn't consistently guaranteed to appear appropriately within the regions outlined by M 𝑜𝑏 𝑗 , thereby causing the unwanted artifacts. To address this challenge, we introduce RCA, which replaces the cross-attention layers in the U-Net to restrict the impact of object-specific words to regions defined by M 𝑜𝑏 𝑗 . Specifically, the latent noisy image is projected to a query matrix q, while the text prompt's embedding is projected to the key k and value v matrices via learned linear projections. The cross-attention maps A ∈ R (ℎ×𝑤 ) ×𝑝 is computed as A = q•k T / √ 𝑑 , where p denote the amount of tokens.Then, certain attention maps A 𝑜𝑏 𝑗 corresponding to object-related tokens (e.g., 'white fox' and 'lemon' in Fig. 4 ) in A are rectified by applying the binary mask M 𝑜𝑏 𝑗 :EQUATIONwhere 𝑎𝑖 𝑗 and 𝑚 𝑖 𝑗 represents the weight of the object-related token's map and the value of M 𝑜𝑏 𝑗 , respectively, at the position (i, j). After that, we obtain the rectified attention maps Â and the output of RCA layer is defined asF =Softmax( Â)v.Through the use of rectified attention maps, this module effectively restricts the impact of object-related words to specific spatial regions on the image features. Consequently, the model can enforce the generation of objects in desired positions and shapes, addressing the coherence problem highlighted earlier.Inspired by the concept of blending two images by separately combining each level of their Laplacian pyramids [5] , our approach involves combining synthesized foreground areas and the given background across different noise levels to maintain the unchanged scene. The underlying principle is that at each step in the diffusion process, a noisy latent is projected onto a manifold of naturally noised images at a specific level. While blending two noisy images (from the same level) may result in output likely outside the manifold, the subsequent diffusion step projects the result onto the next-level manifold, thereby improving coherence [2] .Thus, at each step t, starting from a latent z 𝑡 , we perform a single diffusion step. The resultant latent is segmented based on M 𝑓 𝑔 , yielding a latent denoted z 𝑡 -1,𝑓 𝑔 . In addition, we obtain a noised version of the input background z 𝑏𝑔 * 𝑡 -1 using Equ. 2. The two latents are combined using the foreground mask:EQUATIONTo reinforce the steering effect of the infused prior weights in foreground generation, CFG is extended in each sampling step to extrapolate the predicted noise ε along the direction specified by certain infusions:ε = 𝜀 𝜃 (z 𝑡 |∅) + 𝑠 [𝜀 𝜃 (z 𝑡 |𝑐, 𝑓 ) -𝜀 𝜃 (z 𝑡 |𝑓 ) + 𝜀 𝜃 (z 𝑡 |𝑐, 𝑓 ) -𝜀 𝜃 (z 𝑡 |𝑐)] .Where ∅, c, and f signify a null, caption prompt, and infusion condition, respectively. 𝜀 𝜃 and ε represent the employed LDM and its output noise. The hyperparameter 𝑠 > 0 denotes the guidance scale, and the reinforcement effect becomes stronger as 𝑠 increases.As shown in Fig. 7 , this careful design effectively strengthens the ability to generate more harmonious images, since this leads LDM to become more adept at capturing and preserving the subtle details of the object's appearance and coherence relations. The visualization of the saliency maps derived from our extended CFG is provided in the supplementary. 
experiments 5.1 implementation details: 
 5.1.1 Test benchmark. We employ the only publicly released crossdomain composition benchmark [28] , which contains 332 samples. Each sample consists of a background image, an object image, a foreground mask, an object mask, and a caption prompt. The background images comprise four visual domains: photorealism, pencil sketching, oil painting, and cartoon animation. We adjust all the caption prompts to mark the object-specific words. The details are left to supplementary.We conduct a qualitative comparison between our method and state-of-the-art baselines. The baselines includes Deep Image Blending (DIB) [49] , Blended Diffusion [2] , Paint by Example [46] , SDEdit [29] , and TF-ICON [28] . For the quantitative assessment, all baselines are considered, excluding DCCF, as it is designed for harmonizing images after copy-and-paste operations.Given that most baselines are trained primarily in the photorealism domain, where objective metrics are more effective, we conducted our quantitative comparison specifically in this domain for the fairness. For other domains, we relied on a user study and qualitative comparisons. We utilized the official implementation of all the baselines. Our framework employed the pre-trained Stable Diffusion with the second-order DPM-Solver++ in 20 steps. The hyperparameter 𝛼 for prior weights infusion was set to 0.2 and the scale of classifier-free guidance was set to 2.5 for the photorealism domain and 5 for other cross-domains. LPIPS metric [50] . ( 2) 𝐿𝑃𝐼𝑃𝑆 (𝐹𝐺 ) : evaluates the low-level similarity between the object region and the reference using the LPIPS metric [50] . (3) 𝐶𝐿𝐼𝑃 (𝐼𝑚𝑎𝑔𝑒 ) : assesses the semantic similarity between the object region and the reference in the CLIP embedding space [49] .(4) 𝐶𝐿𝐼𝑃 (𝑇 𝑒𝑥𝑡 ) : measures the semantic alignment between the text prompt and the resultant image [33] .The qualitative comparison results in Fig. 6 highlight the superior performance of PrimeComposer in seamlessly integrating objects across various domains while preserving their appearance and synthesizing natural coherence. Notably, some baselines, such as Paint by Example, face challenges in maintaining the appearance of the given object. Blended Diffusion, relying solely on text prompts, fails to ensure the synthesized object matches the reference image. Additionally, Deep Image Blending, DCCF and SDEdit struggle to achieve harmonious transitions. While TF-ICON performs relatively well compared to other methods, it still faces difficulties in simultaneously preserving object appearance and synthesizing natural coherence. For example, it fails to preserve the identity features of the tortoise in the first row and struggles to achieve optimal coherence around the synthesized building in the last row. Additional qualitative comparison results are left to supplementary.As demonstrated in Tab. 1, our proposed PrimeComposer outperforms all competitors consistently across all metrics, highlighting the exceptional visual quality of the composite images it generates. Notably, TF-ICON also exhibits commendable generation quality. However, its composite self-attention map, derived from multiple samplers, introduces synthesis confusion. As a consequence, the realism and harmony of its resulting images are compromised. In contrast, our PrimeComposer Competently alleviates these challenges. Notably, PrimeComposer surpasses it by 1.85 and 2.15 on 𝐶𝐿𝐼𝑃 (𝐼𝑚𝑎𝑔𝑒 ) and 𝐶𝐿𝐼𝑃 (𝑇 𝑒𝑥𝑡 ) , respectively. This demonstrates that our method outperforms previous approaches in preserving object appearance and achieving harmonious coherence synthesis.We conduct a comparison with the previous SOTA training-free method on an NVIDIA A100 40GB PCIe. Considering the time depends on the size of the user mask and reference image, we measure the averaged inference time per image across various domains in the test benchmark to ensure fairness. As shown in Tab. 2, PrimeComposer consistently exhibits faster inference times than TF-ICON across all domains. Remarkably, our inference time in the photorealism domain is notably lower than TF-ICON, specifically achieving a time of 14.32 seconds. This phenomenon is expected, considering that TF-ICON employs four samplers for composition, while our approach only utilizes two samplers, i.e., the pre-trained LDM and a Correlation Diffuser, during the process, resulting in a more efficient computational performance. See supplementary for additional inference speed comparisons with training baselines, which further proves the efficiency of our method.We conduct a user study to compare image composition baselines across different domains. Specifically, we invited 30 participants we conduct ablations on key design choices in the following cases:(1) Baseline, where the composition is synthesized by LDM from T to 0 only with the caption prompt. The initial noise is the pixel composition derived from inverted codes of the given object and background; (2) Background combining is applied to maintain the unchanged scene; (3) Correlation infusion is employed to steer the natural coherent relation establishment; (4) Object infusion is employed to steer the preservation of the object's appearance;(5) Region-constrained Cross-Attention is used to enforce the generation of objects in desired positions and shapes, addressing the coherence problem caused by the caption prompt; (6) Extended CFG, tailored to reinforce the steering impact of prior weights infusion.Quantitative results: Tab. 3 presents quantitative ablation results, showcasing the superior performance of our complete algorithm across all metrics except 𝐶𝐿𝐼𝑃 (𝑇 𝑒𝑥𝑡 ) . It is noteworthy that the baseline achieves the best 𝐶𝐿𝐼𝑃 (𝑇 𝑒𝑥𝑡 ) result, as it generates compositions solely relying on the caption prompt without any additional constraints [28] . These ablation results underline the effectiveness of our proposed algorithm in enhancing various aspects of the composition process. Qualitative results: To further visualize the effectiveness of each design choice, we provide qualitative results shown in Fig. 7 . These results directly prove the indispensable role of all design choices. More qualitative ablation results are left to supplementary. 
conclusion: 
 In this paper, we formulate image composition as a subject-guided local image editing task and propose a faster training-free diffuser, PrimeComposer. Leveraging well-designed attention steering, primarily through the Correlation Diffuser, our method seamlessly integrates foreground objects into noisy backgrounds while maintaining scene consistency. The introduction of Region-constrained Cross-Attention further enhances coherence and addresses unwanted artifacts in prior methods. Our approach demonstrates fastest inference efficiency and outperforms existing methods both qualitatively and quantitatively in extensive experiments. 
: 
 of appearance information. These issues worsen with their excessive focus on background generation, even when unnecessary in this task. This not only impedes their swift implementation but also compromises foreground generation quality. Moreover, these methods introduce unwanted artifacts in the transition area. In this paper, we formulate image composition as a subject-based local editing task, solely focusing on foreground generation. At each step, the edited foreground is combined with the noisy background to maintain scene consistency. To address the remaining issues, we propose PrimeComposer, a faster training-free diffuser that composites the images by well-designed attention steering across different noise levels. This steering is predominantly achieved by our Correlation Diffuser, utilizing its self-attention layers at each step. Within these layers, the synthesized subject interacts with both the referenced object and background, capturing intricate details and coherent relationships. This prior information is encoded into the attention weights, which are then integrated into the self-attention layers of the generator to guide the synthesis process. Besides, we introduce a Region-constrained Cross-Attention to confine the impact of 
 
 Appendix:  


Decision: The paper describes a 2D composition framework based on diffusion models for reference-based and cross-domain editing that integrates foreground objects into noisy backgrounds.

All reviewers share a positive judgment about the paper (WA, BA,BA). The rebuttal convincingly addressed initial concerns.

The paper is of good quality, carefully identifies the weaknesses from prior methods and proposes solid improvements. Experiments are evaluated solid and showing state of the art performance. Qualitative results are good.

Most of the limitations raised by reviewers were solved in the rebuttal. Some minor concerns remain, such as the fact that only foreground objects can be modeled. 

Overall, the positive aspects overcome negative ones. The AC tends to agree with the reviewers and recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes PrimeComposer, a 2D composition framework for reference-based local editing and cross-domain editing. To improve editing quality and inference speed, this framework involves several mechanisms such as attention steering, Region-constrained Cross-Attention and extended classifier-free guidance.
strength_weakness: strength_weakness: - The paper carefully identifies the weaknesses from prior methods (mainly TF-ICON) such as identity preservation, background preservation and style inconsistency. The technical components from the paper are closely related to these weaknesses and introduce solid improvements.
- Compared to TF-ICON, PrimeComposer is more stable in performance and its design of the pipeline is more straightforward.
limitations: limitations: - Though the RCA will reduce the unwanted editing in the background, it will also limit the pose change or global effects (such as shadow or reflection) of the inserted object. Maybe this can be improved by optionally dilating the mask M_obj?
- To solve the coherence problem, the proposed method restricts weight infusion only in the decoder part of the unet. This seems like a trade-off between diversity and consistency. Can more control be added here (like how many layers to inject the weights) to enable more editing flexibility?
- In the task of composition, sometimes there should be geometry alignment on the inserted object to have a natural composite image. In Fig 5, the croissant looks floating in the air; in Fig 6, the turtle looks more like copy-paste than the result from TF-ICON. This will limit the application of the proposed method.
- The paper should have more quantitative metrics to evaluate the overall quality of the generated images, such as FID and Quality Score.
paper_summary: paper_summary: This paper considers image composition based on diffusion models and models it as a subject-guided local editing problem focusing only on foreground generation. 
Specifically, this paper proposes an efficient, training-free method, denoted as PrimeComposer, which uses attention steering to fuse a given object into an image. 
The quantitative and qualitative results have demonstrated the advantages of the proposed PrimeComposer in terms of both performance and efficiency.
strength_weakness: strength_weakness: 1. The proposed method is totally training-free, that is, it requires no training at all, which is friendly in terms of computing resources, and the composition results are naturally consistent in style and details.

2. Sufficient qualitative and quantitative experimental results strongly support the excellent performance and efficiency of the proposed PrimeComposer, and the experimental results are impressive.

3. The effectiveness of each module of the proposed method is also verified by sufficient ablation studies.

4. The content of this paper is compatible with the theme of Multimedia.
limitations: limitations: 1. The innovation of the proposed method and the insight conveyed are limited. Adjusting attention to enable the generation with a specific object has been widely used in image-driven generation, personalized generation and other fields. In addition, controlled generation by restraining attention maps has also been relatively mature in image editing and other fields.

2. The proposed method is limited to the fusion of foreground objects, and from the results presented in the paper, it is still limited to one single object, and there is a lack of examples of multi-object composition. The composition process is more like a simple copy and paste + style transfer, lack of object motion or shape/texture changes.

3. There are some minor suggestions on the writing of the paper, such as the Contributions paragraph in the Introduction section. It is more appropriate to use the full name of the proposed modules to avoid readers' forgetting, such as Correlation Diffuser (CD). There is still room for improvement in the writing of the paper to make it easy to understand and follow.

4. The details in Figure 3 are very sloppy, such as the position of arrows in the figure, the size of words and symbols, etc., which should be considered for further improvement.

5. Similar to the strategy mentioned in the paper, some works also attempt insert specific objects into new scenes through Attention, which may be worth considering to be included in Related Works, such as ViCo [1*] and Intelligent Grimm [2*].

[1*] ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generation

[2*] Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models
paper_summary: paper_summary: 1. The paper is about a faster training-free diffuser called PrimeComposer, which is used for image composition. The authors propose a method that integrates foreground objects into noisy backgrounds while maintaining scene consistency. 
2. The paper leverages attention steering, primarily through the Correlation Diffuser, to guide the synthesis process. It also introduce Region-constrained Cross-Attention to enhance coherence and address unwanted artifacts. 
3. PrimeComposer demonstrates faster inference efficiency and outperforms existing methods in extensive experiments.
strength_weakness: strength_weakness: 1. PrimeComposer introduces a faster progressively combined diffusion approach for image composition with attention steering. This approach is different from previous methods, which allows for more efficient and effective image composition.
2. PrimeComposer utilizes a composite self-attention map derived from multiple samplers. This approach helps to preserve object appearance and achieve harmonious coherence synthesis in the resulting images. 
3. PrimeComposer has been evaluated and compared with other competitors. The quantitative analysis shows that PrimeComposer consistently outperforms all competitors across all metrics, highlighting the exceptional visual quality of the composite images it generates. 
4. Furthermore, the paper provides clear explanations and descriptions of the proposed approach, making it easy to understand and replicate.
limitations: limitations: 1.  The Correlation Diffuser, as the core innovation in this paper, establishes the correlations between the synthesized object and background, as well as utilizes the object appearance features to guide the self-attention maps. However, how is the hyperparameter α determined for prior weights infusion?
2. The absence of a trained model in the supplementary document raises concerns about how to conduct testing, thereby prompting doubts regarding the technical accuracy and efficacy of the proposed method.