TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 19:48:27

Abstract: We present TALE, a novel training-free framework harnessing the power of text-driven diffusion models to tackle cross-domain image composition task that aims at seamlessly incorporating user-provided objects into a specific visual context regardless of domain disparity. Previous methods often involve either training auxiliary networks or finetuning diffusion models on customized datasets, which are expensive and may undermine the robust textual and visual priors of pretrained diffusion models. Some recent works attempt to break the barrier by proposing training-free workarounds that rely on manipulating attention maps to tame the denoising process implicitly. However, composing via attention maps does not necessarily yield desired compositional outcomes. These approaches could only retain some semantic information and usually fall short in preserving identity characteristics of input objects or exhibit limited background-object style adaptation in generated images. In contrast, TALE is a novel method that operates directly on latent space to provide explicit and effective guidance for the composition process to resolve these problems. Specifically, we equip TALE with two mechanisms dubbed Adaptive Latent Manipulation and Energy-guided Latent Optimization. The former formulates noisy latents conducive to initiating and steering the composition process by directly leveraging background and foreground latents at corresponding timesteps, and the latter exploits designated energy functions to further optimize intermediate latents conforming to specific conditions that complement the former to generate desired final results. Our experiments demonstrate that TALE surpasses prior baselines and attains state-of-the-art performance in image-guided composition across various photorealistic and artistic domains.

Main paper: 
 introduction: 
 Image composition, as a branch of image editing, has progressively garnered attention in recent years [3, 25, 29, 42, 49, 52] . Typically, this task involves integrating a user-specified image or text prompt into a specified area of background while ensuring that the composited image appears natural and seamless, exhibiting consistent lighting conditions and a smooth foreground-background transition. Image composition has been employed in various fields. For example, the entertainment industry relies on image composition to create stunning visual effects, facilitating the seamless integration of actors and objects into fantastical environments that would be impractical or impossible to capture in real life. Moreover, image composition can also be used in interior design. Specifically, it is used to place virtual furniture into real interior spaces, aiding in visualization and decision-making processes for both designers and clients. In light of these significant and useful applications, it is imperative to explore the field of image composition fully. The prevailing methods for image composition involve finetuning pre-trained models with customized datasets, aiming to improve the semantic coherence of the composited results. For instance, Paint by Example [49] utilizes object detection and data augmentation to generate pairs consisting of a foreground and a background. These pairs are used for training the diffusion model. AnyDoor [3] designs an identity (ID) extractor module to distill the characteristic features of specified objects. These extracted features are subsequently employed as conditional inputs to guide the training process of the diffusion model. While these trainingbased methods have demonstrated remarkable performance, they require substantial computational effort that limits the accessibility for researchers with constrained resources. Training-free methods (e.g., TF-ICON [29] ) offer a promising direction by injection mechanism to merge the self-attention maps of foregrounds and backgrounds. However, they still face critical challenges, particularly in preserving the identity of the composited elements. Besides, they demonstrate subpar performance when tackling cross-domain image composition.To mitigate these drawbacks, we present TALE, a training-free framework harnessing the power of text-driven diffusion models to tackle cross-domain image composition task, aiming at seamlessly incorporating user-provided objects into a specific visual context regardless of domain disparity. Specifically, TALE functions in the latent spaces, offering precise and potent direction within the compositing workflow to remedy the above-mentioned issues. TALE is equipped with two distinct components: Adaptive Latent Manipulation and Energy-guided Latent Optimization. The former establishes an initial noisy latent conducive to beginning the composition, then applies normalization to iteratively guide subsequent composing steps. In complement, the latter utilizes specific energy functions to further refine the normalized intermediate latents. This synergistic mechanism ensures the production of the intended visual outcomes. The experimental results and user studies reveal that the proposed TALE outperforms existing methods. The code will be made available to promote future research.Overall, our contributions are listed as follows:• We propose TALE, a novel training-free framework capable of seamlessly incorporating user-provided objects into diverse visual contexts across multiple domains. • TALE excels in preserving the identity characteristics of input objects while harmonizing their style with the backgrounds, resulting in highly realistic and aesthetically pleasing composited images thanks to its Adaptive Latent Manipulation and Energy-guided Latent Optimization mechanisms. • Extensive experiments and user studies provide compelling evidence of TALE's strength over prior work. We will release the code to promote future research. 
related work 2.1 image composition: 
 Image composition is an essential task utilized in various image editing platforms. The primary goal is to integrate an object into a given background [32] . The composition models should create a visually seamless and convincingly realistic composited image, making it imperceptible for observers to discern any traces of manipulation. Generally, image composition can be categorized into two types based on whether the original object's structure or contour is preserved. When structure preservation is necessary, some works design image harmonization techniques [4, 11, 12, 26, 45, 46] , emphasizing color consistency and luminance coherence across the composited areas. Other methods introduce image blending strategies [1, 28, 47, 54] to remedy the unnatural boundaries between the foreground and background, ensuring a seamless integration while maintaining the integrity of the original structure.Another line of work suggests that maintaining the object's identity is sufficient, allowing for changes in its perspective and enabling more flexibility [3, 21, 24, 25, 29, 42, 43, 49, 52] . For instance, Paint by Example [49] leverages object detection and data augmentation techniques to create foreground-background pairs, with the augmented foreground image acting as a conditioning in the training of a diffusion model. AnyDoor [3] incorporates an ID extractor to capture the identity features of given objects, which are utilized as conditions for training the diffusion model. It is worth mentioning that TF-ICON [29] introduces a training-free framework, taking advantage of pre-trained text-to-image models for image composition. In particular, it incorporates the self-attention maps extracted when reconstructing foregrounds and backgrounds to melt them together. It is shown that the performance of TF-ICON surpasses existing image composition methods in versatile visual domains, yet they struggle to preserve object identity features and suffer from incohesive style adaptation.Generally, the proposed TALE adheres to a training-free routine but distinguishes itself from TF-ICON in that our method is capable of well preserving the object identity and seamlessly blending to diverse domains of different styles, powered by the proposed Adaptive Latent Manipulation and Energy-guided Optimization mechanisms.In recent years, diffusion models [2, 10, 13, 14, 33, 35, 38, 40, 53, 57] have become the mainstream of generative models across various domains, owing to their exceptional fidelity and diversity in generated results when compared with GANs [9] and VAEs [19] .Notably, the Latent Diffusion Model (LDM) [37] performs the diffusion process in a VAE-compressed latent space, thereby improving computational efficiency. DDIM [40] introduces a novel approach to accelerate the latent diffusion processes. Remarkably, DDIM inversion has been effectively utilized for editing purposes and has been integrated into other image composition methods [29] . Imagen [38] introduces multiple diffusion models for progressive generation, enhancing the resolution of generated images step by step. SD-XL [35] enlarges the model size and designs curated strategies to enhance the image quality. DiT [33] utilizes Transformers as the backbone and proves the scaling ability. ControlNet [53] introduces an additional branch to receive the additional conditions, such as the canny maps and segmentation maps. Uni-ControlNet [57] enables processing multiple conditions at the same time. Typically, these methods require training or finetuning on the additional conditions involved to enable a certain degree of control over specific tasks.To enable controllable generation with different conditions at sampling time, several methods leverage energy functions to guide the diffusion process [6, 7, 22, 50, 56] , alleviating the cost of training. In particular, EGSDE [56] introduces a time-dependent energy function designated for unpaired image-to-image translation task. Differently, FreeDom [50] proposes a flexible time-independent formulation for energy functions that facilitate different image editing tasks on multiple conditions. 
preliminary 3.1 latent diffusion model: 
 We leverage the pre-trained text-to-image LDM for our composition model. The diffusion procedure follows the standard formulation in [13, 39, 41] , which comprises a forward diffusion and a backward denoising process. Given a data sample x ∼ 𝑝 (x), an autoencoder consisting of an encoder E and a decoder D will first project it into latent z 0 = E (x). Subsequently, the diffusion and denoising processes are conducted in latent space. Once the denoising is finished and a final clean latent ẑ0 is generated, the sample can then be decoded via x = D (ẑ 0 ).The original diffusion models [13] can only serve as an unconditional generator. In order to control the generation process with a desired condition 𝑐, classifier-guided methods [5, 27, 31, 56] propose to alter the prediction of the denoising network as:EQUATIONwhere 𝜎 𝑡 is predefined diffusion scalar and 𝜙 is a trained timedependent noisy classifier that estimates the label distribution of each sample of z 𝑡 . The term ∇ z 𝑡 log 𝑝 𝜙 (c|z 𝑡 ) can be interpreted as a correction gradient that steers z 𝑡 toward a hyperplane in the latent space where all latents are compatible with the given condition c.To approximate such a gradient, a flexible and straightforward way is utilizing the energy guidance function [22, 50, 56] as follows:EQUATIONHere 𝜉 (z 𝑡 , 𝑡, c) denotes an energy function that quantifies the compatibility between the condition c and the noisy latent z 𝑡 . The more z 𝑡 conforms to c, the smaller the energy value should be. Such a loose property enables great flexibility in designing suitable 𝜉 to suit for each condition c. Correspondingly, the updated conditional backward process can be written as:EQUATIONwhere z 𝑡 -1 ∼ 𝑝 𝜃 (z 𝑡 -1 |z 𝑡 ) and 𝜌 𝑡 is a scale factor. We base on this equation to derive a latent optimization mechanism to modulate the composition process. 
method 4.1 problem formulation: 
 Given a background (main) image x 𝑏𝑔 , a foreground (object) image x 𝑓 𝑔 with segmentation mask M 𝑜𝑏 𝑗 , a text prompt P, and a userprovided binary mask M 𝑢 indicating the region of interest within x 𝑏𝑔 , the objective of cross-domain image composition is to generate composited image x 𝑟𝑒𝑠 that harmoniously acquires three properties. Firstly, the inputted object appears in the masked region of x 𝑟𝑒𝑠 and picks up a similar style to x 𝑏𝑔 while preserving its identity features, i.e. 𝐼𝐷 (x 𝑟𝑒𝑠 ⊙ M 𝑢 ) ≈ 𝐼𝐷 (x 𝑓 𝑔 ) and 𝑆𝑡𝑦𝑙𝑒 (x 𝑟𝑒𝑠 ⊙ M 𝑢 ) ≈ 𝑆𝑡𝑦𝑙𝑒 (x 𝑏𝑔 ). Secondly, the complementing background area of x 𝑟𝑒𝑠 closely resembles the corresponding area of x 𝑏𝑔 , i.e. point for composing byx 𝑟𝑒𝑠 ⊙ (1 - M 𝑢 ) ≈ x 𝑏𝑔 ⊙ (1-M 𝑢 ).EQUATIONwhere z ∼ N (0, I), M 𝑧 𝑏𝑔 = 1 -M 𝑧 𝑢 indicates region outside M 𝑧 𝑢 , andM 𝑧 𝑡𝑟𝑎𝑛 = M 𝑧 𝑢 ⊕ M 𝑧 𝑜𝑏 𝑗represents the transition area. Note that these masks are correspondingly rescaled to latent resolution from those mentioned in Section 4.1. After inversion stage, composition is essentially a backward process which involves concurrently denoising z 𝑏𝑔 𝑇 , z 𝑓 𝑔 𝑇 , and z 𝑟𝑒𝑠 𝑇 . The incorporation in Eq. 4 is applied at initial timestep 𝑇 while for 𝑡 < 𝑇 , the composition process is implicitly controlled by injecting self-attention maps obtained when denoising z 𝑏𝑔 𝑡 and z 𝑓 𝑔 𝑡 into those of z 𝑟𝑒𝑠 𝑡 in specific manner [29] . Though self-attention maps could bring about some semantic information of the inputted object to the resulting image, they are susceptible to identity features loss and incohesive style adaptation, as illustrated in Fig. 3 . Moreover, randomly initializing values within transition area M 𝑧 𝑡𝑟𝑎𝑛 can produce unwanted artifacts. To overcome these issues, we aim to induce explicit guidance that directly leverages noisy latents to capture identity features better while seamlessly altering domain style. Our empirical observations reveal that it can be achieved by initiating the composition process at a later timestep instead of 𝑇 . Formally, we select timestep 0 < 𝑇 ′ < 𝑇 and employ z 𝑟𝑒𝑠 𝑇 ′ as the starting point for composing:EQUATION)The rationale behind preference of 𝑇 ′ over 𝑇 is the more the denoising progresses, the more style and identity information are reconstructed in z 𝑇 ′ 𝑓 𝑔 and z 𝑇 ′ 𝑏𝑔 comparing to those from timestep 𝑇 , hence the more informative and effective they can be brought to z 𝑇 ′ 𝑟𝑒𝑠 . Moreover, the pre-trained denoising network 𝜖 𝜃 can retain the layout structure of z 𝑇 ′ 𝑟𝑒𝑠 while gradually rectifying its texture throughout the remaining duration 𝑡 ∈ [0,𝑇 ′ ). Therefore, commencing the composition process at timestep 𝑇 ′ with z 𝑇 ′ 𝑟𝑒𝑠 explicitly leads to desired outcomes without any intervention into self-attention features. Conceivably, this shares a similar intuition with SDEdit [30] to hijack the reverse denoising process, but while SDEdit firstly performs composition in pixel space and then perturbation, we adopt a reversed manner by conducting inversion before composition, allowing for better style harmonization while effectively preserving background and foreground contents. We discuss how to choose the appropriate value for 𝑇 ′ in Section. 5.3. Adaptive Latent Normalization. For challenging cases where a significant domain discrepancy exists between x 𝑏𝑔 and x 𝑓 𝑔 , although the Selective Initiation operation is able to integrate identity information of the input object into the composited image, its color hue falls short of the anticipated outcome. For instance, when x 𝑏𝑔 is black-and-white but x 𝑓 𝑔 is colorful, as in Fig. 6 , some colors are where 𝜇 and 𝜎 denote channel-wise means and standard deviations. Besides, we introduce a threshold 𝜆 𝑡 to further balance the contentstyle trade-off of the modulated latent as:EQUATIONFinally, substituting z𝑎𝑑𝑛 𝑡 into z 𝑟𝑒𝑠 𝑡 results in the updated z𝑟𝑒𝑠 𝑡 that can preserve content information of object region while its color tone is gradually aligned better with the background.Energy Function Design. Despite capturing object identity features and emulating the style of the background, the resulting z 𝑟𝑒𝑠 𝑡 might be inconsistent with the contextual guidance provided by the input text prompt P. This may undermine the rich semantic prior of diffusion model 𝜖 𝜃 and eventually lead to deviation from intended outcomes similar to TF-ICON. Inspired by [34, 48, 50] , we propose to leverage the updated conditional denoising process in Eq. 3 and design suitable energy function 𝜉 to further optimize z 𝑟𝑒𝑠 𝑡 conforming with P. Specifically, given latent variable z 𝑟𝑒𝑠 𝑡 at timestep 𝑡 ∈ [0,𝑇 ′ ), we first derive the composited image x 𝑟𝑒𝑠 0|𝑡 from z 𝑟𝑒𝑠 𝑡 and predicted noise ε𝑡 viaEQUATIONwhere ε𝑡 = 𝜖 𝜃 (z 𝑟𝑒𝑠 𝑡 , 𝑡) and D is the decoder mapping from latent back to image space. With such clean prediction on image space, we can then employ external models pre-trained on normal data to estimate 𝜉 (z 𝑟𝑒𝑠 𝑡 , 𝑡, P) as below: 𝜉 (z 𝑟𝑒𝑠 𝑡 , 𝑡, P) ≈ F = 1cos(EMB P (x 𝑟𝑒𝑠 0|𝑡 ), EMB P (P)).Here EMB P projects input into an aligned embedding space via pre-trained multimodal projector P, and F denotes a distance measuring function, which is one minus cosine similarity between two embedding vectors. The obtained distance then serves as a global penalty to backpropagate the computational graph and obtain a x 𝑟𝑒𝑠 0|𝑡 = D ((z 𝑟𝑒𝑠 𝑡 -𝜎 𝑡 ε𝑡 )/𝛼 𝑡 )4:F = 1cos(EMB P (x 𝑟𝑒𝑠 0|𝑡 ), EMB P (P))5:F ′ = ||G P (x 𝑟𝑒𝑠 0|𝑡 ⊙ M 𝑢 ) -G P (x 𝑏𝑔 )|| 2 𝐹 6: z𝑟𝑒𝑠 𝑡 = z𝑟𝑒𝑠 𝑡 -(𝜂∇ z 𝑟𝑒𝑠 𝑡 F + 𝜂 ′ ∇ z 𝑟𝑒𝑠 𝑡 F ′ ) ⊙ M 𝑧 𝑜𝑏 𝑗 7: end for 8: ẑ𝑟𝑒𝑠 𝑡 -1 = z𝑟𝑒𝑠 𝑡 ⊙ M 𝑧 𝑜𝑏 𝑗 + z 𝑏𝑔 𝑡 ⊙ (1 -M 𝑧 𝑜𝑏 𝑗 ) 9: return ẑ𝑟𝑒𝑠 𝑡 -1gradient on z 𝑟𝑒𝑠 𝑡 . By incorporating Eq. 3 and Eq. 9, we can derive the updated composition process as:EQUATIONin which 𝜂 serves as the learning rate of each optimization step. We leverage CLIP [36] model with powerful text-image alignment capability as the projector.Note that 𝜉 can be approximated by a combination of multiple distance functions, one can also compute the distance of the style information between 𝑥 𝑟𝑒𝑠 0|𝑡 within M 𝑢 (the object patch) and 𝑥 𝑏𝑔 to attain better local style cohesion:EQUATIONwhere G denotes the Gram matrix [17] of the feature map obtained from the projector P that captures the second-order style information. This extra regularization can be added to Eq. 10 as:EQUATION)Since object area is the region to be edited while background must remain unchanged, it is intuitive to only optimize the object patch within M 𝑧 𝑜𝑏 𝑗 using Eq. 12, while background region outside the mask can be effectively maintained via replacement trick as in [29] .Timestep Constraint. It is observed that applying normalization and optimization for every timestep 𝑡 ∈ [0,𝑇 ′ ) may lead to noticeable artifacts in transition area. Thus, similar to [1, 29] , we introduce threshold 𝜏 to regulate them within 𝑡 ∈ [𝑇 ′ -𝜏,𝑇 ′ ) only, allowing sufficient time left for diffusion model to rectify the outputs. 
experiments 5.1 experimental setups: 
 Baseline Benchmark. We utilize the benchmark dataset provided by the TF-ICON [29] for evaluation of our method. It includes 332 samples, each comprising a background image, an object image, a user-provided mask, an object segmentation mask, and a text prompt. The background images are divided into four visual domains: photorealism, pencil sketching, oil painting, and cartoon animation. The object images comprise more than 60 categories from photorealism domain with segmentation masks obtained using SAM [20] model. The text prompts are manually annotated according to the semantics of background and object images. Extended Dataset. Since the baseline benchmark is heavily skewed towards the photorealism domain with over 70% of samples and provides a limited number of background images for assessment, we propose an extended dataset with more non-photorealistic samples and diverse backgrounds. We randomly select artistic domain images from Clipart1k, Watercolor2k, and Comic2k [16] , to be background images, utilizing their object bounding box annotations for user-specified mask generation. For each background, we randomly select an object of class [CLS] and adopt BLIP2 [23] model to generate caption of template "A [CLS] and . . . ". Then, we leverage Inpaint Anything [51] framework to inpaint the selected object location, obtaining a clean background image. Besides, object images are sampled from the baseline benchmark due to their category diversity. Subsequently, we pair the object and background images, and accordingly replace [CLS] in the background caption with the category [CLS*] of the paired object. Lastly, we manually remove unreasonable pairs for sanity and eventually obtain an extended benchmark of 207 high-quality non-photorealistic domain samples with diverse backgrounds for evaluation, complementing what is lacking from the baseline. Implementation Details. We first adopt the preprocessing pipeline from TF-ICON [29] to preprocess each data sample so that the input object is rescaled and relocated to correspond with userinputted mask. In addition, we employ Inpaint Anything [51] model to remove unwanted objects underneath the user mask to produce a clean background image for composing. Then, we conduct composition processes using our proposed training-free approach TALE of which the overall framework is depicted in Fig. 2 . Specifically, we leverage the inversion technique introduced in [29] to invert background and foreground images into latent representations z 𝑏𝑔 𝑇 and z 𝑓 𝑔 𝑇 then iteratively denoise them for 𝑇 = 20 timesteps while conducting composition process intertwine starting from 𝑇 ′ = 8. Subsequently, we proceed to normalize and optimize the intermediate composited latent z 𝑟𝑒𝑠 𝑡 via our proposed Adaptive Latent Normalization (Algorithm 1) and Energy-guided Latent Optimization (Algorithm 2) operations with 𝜏 = 5. We respectively set 𝜆 𝑡 = 0.1 + 0.5(𝑇 ′ -𝑡)/𝜏 for normalization and 𝑁 = 3, 𝜂 = 15, 𝜂 ′ = 0.15 for optimization. We fix the random seed for fair comparisons and conduct all experiments on NVIDIA Geforce RTX 3090 GPUs, where the composition takes about 23 seconds per sample, depending on the size of the foreground image and user mask. Note that these settings are kept by default for every cross-domain experiments, and for same-domain composition, we adjust 𝑇 ′ = 6, 𝜏 = 3, 𝜆 𝑡 = 0.1 and skip optimization as domain discrepancy between background and foreground images is negligible.We compare TALE with prior SOTA and concurrent works that are capable of performing image-guided composition, including TF-ICON [29] , SDEdit [30] , Blended Diffusion [1] , Paint by Example [49] , AnyDoor [3] , ControlCom [52] , and ObjectStitch [42] . Qualitative Results. Qualitative results shown in Fig. 4 highlight the superiority of our method across all domains. First, TALE generates high-quality composited images of which the objects are stylized according to target backgrounds more naturally. Second, the identity features of input objects are better preserved. Third, the complementing background regions of composited images remain unchanged. Fourth, the objects seamlessly blend into the backgrounds without noticeable artifacts in the transition area. In one hand, although AnyDoor, ControlCom, and ObjectStitch can compose images within their photorealistic training domain, they suffer from poor adaptation to other domains. On the other hand, TF-ICON and Paint by Example can provide certain degree of freedom for composing in different domains yet they fall short in retaining object identities and altering color style. For SDEdit and Blended Diffusion, while the former often causes unwanted changes to the background, the latter solely resorts to text prompt for composing; hence, its results tend to deviate from user's intention. Quantitative Results. We first follow the prior works to perform quantitative comparisons using four metrics: LPIPS 𝑏𝑔 [55] to assess background preservation, LPIPS 𝑓 𝑔 [55] to measure lowlevel similarity between foreground image and the edited region, CLIP 𝐼𝑚𝑎𝑔𝑒 [36] to examine the semantic correspondence between foreground image and the edited region in CLIP embedding space, and CLIP 𝑇 𝑒𝑥𝑡 [36] to evaluate the semantic alignment between input text prompt and the composited image. However, since these metrics do not assess domain style adaptability and are known for texture and semantic bias [8, 18] in which style information can affect the scores, we only employ them for evaluating composition within the same photorealism domain. As demonstrated in Tab. 1, our method TALE achieves the best performance among trainingfree approaches, even outperforms several frameworks that are trained on this domain.For cross-domain comparisons, we adopt the recent evaluation protocol from [18] , which can precisely examine domain transferability in terms of style and content similarity. Specifically, we leverage their pre-trained discriminator to predict color style similarity score between the edited patch of composited image and the background. For content similarity, we utilize LDC [44] model to extract edge features of background, foreground, and composited images. These features are more tolerant of style changes and hence can be used to assess content preservation. We then compute content similarity score with a slight modification asEQUATIONwhere SSIM 𝑓 𝑔 denotes SSIM calculated between edge features of foreground image and edited region of resulting image, and SSIM 𝑏𝑔 is calculated on the complementing background area. This metric formulation can effectively reflect both background and object identity preservation capabilities. Results presented in Fig. 5 show that we attain the most balanced content-style trade-off across all domains. We can observe that although Anydoor and ControlCom have high content similarity scores, they often fail to alter the object style. In contrast, SDEdit may obtain high style similarity scores yet they struggle to retain content information. User Study. To subjectively evaluate the performance of our TALE compared to other methods, we invite 50 users to participate in a user study. We show each of them 20 to 30 image sets randomly selected from a pool of 310 questions each consists of a background image, a foreground image, and two composited options of which one is from ours and the other is randomly picked from 7 results generated by prior works. Users are required to select the bettercomposited image based on comprehensive criteria considered foreground content-style balance, background preservation, text alignment, and seamless composition. After collecting user responses, we computed the average preference percentage of our method over others. Fig. 7 shows that TALE is greatly favored by the users.Component Effectiveness. We sequentially ablate the key elements of our proposed TALE on the extended dataset with the following configurations: (1) Baseline, in which the composition is generated by a plain denoising process from 𝑇 to 0 with neither adaptive latent manipulation nor energy-guided optimization. The initial point is composed by incorporating inverted noises at 𝑇 ′ = 𝑇 ; (2) 𝑇 ′ is selectively set; (3) The adaptive normalization is additionally conducted; (4) The energy-guided optimization is 𝑇 ′ Selection. Intuitively, the more the denoising progresses, the more information about backgrounds and objects are reconstructed, hence the more effectively they can be composed into final outcomes. To validate this intuition, we experiment with the influence of different choices of 𝑇 ′ on the extended dataset. Consistent results are demonstrated in Fig. 8 and Tab. 3. Notably, too large 𝑇 ′ leads to content information loss, while too small 𝑇 ′ affects domain style adaptation. 
conclusion: 
 We have presented a novel training-free framework dubbed TALE leveraging powerful text-driven diffusion models for high-quality cross-domain image-guided composition. TALE is equipped with two components, namely Adaptive Latent Manipulation and Energyguided Latent Optimization, that works in synergy to construct and control the composition process, seamlessly incorporating userprovided objects into a specific visual background of different domains. Our experimental results highlight the superiority of our approach over prior and concurrent works, achieving state-of-the-art performance. We hope that our method can inspire future research on similar or relevant topics. 
 
 Appendix:  


Decision: In this paper the author present a novel training-free framework harnessing the power of text-driven diffusion models to tackle a cross-domain image imposition task that aims at seamlessly incorporating user-provided objects into a specific visual context. Reviewers appreciate the comprehensive evaluation and the quality of the results. There were some issues raised and after rebuttal they have been mostly resolved and three of the reviewers give it a borderline accept and the fourth raised it to weak accept.

Reviews: review: paper_summary: paper_summary: This work introduces a training-free framework, TALE, to incorporate objects into a visual context. TALE is comprised of two stages, Adaptive Latent Manipulation and Energy-guided Latent Optimization. Adaptive latent manipulation starts denoising at $T'$, where $0<T'<T$, and composes the object and the background directly with object mask and without transition area mask. Adaptive latent manipulation also introduces a latent normalization to reduce the significant domain discrepancy exists between object and background. Energy-guided latent optimization introduces CLIP as a pre-trained multimodal projector $\mathscr{P}$ to measure the similarity between predicted image and prompt as extra guidance. Comprehensive experiments show effectiveness of TALE.
strength_weakness: strength_weakness: - The overall paper is easy to read and clear.
- Comprehensive experiments have been done to show effectiveness. 
- Method is training free.
limitations: limitations: - The method only showcases single object incorporation cases. Multiple objects case might be interesting to see. 
- The prompts guidance are plain description of object and background, which are easy cases without interactions. And the output looks like style-based copy paste. More challenging cases will be interesting to see.
paper_summary: paper_summary: This work proposes TALE for image composition. TALE features adaptive latent manipulation and energy-guided optimization in a training-free manner. Experimental results show that TALE outperforms previous methods, especially in cross-domain cases.
strength_weakness: strength_weakness: 1. TALE can handle cross-domain image composition, where many previous methods failed.
2. TALE does not need training before composing images.
3. The performance of TALE seems good.
limitations: limitations: ### Major issues
1. It would be better if the authors could further clarify the denoising schedule. Does TALE adopt an accelerated denoising process like DDIM? Does $T = 20$ in Line 693 correspond to the full noise at (conventionally) time step 1000?
2. The adaptive latent normalization in Sec. 4.2 operates on latent space. However for Stable Diffusion (since the authors did not specify which pretrained LDM was used in TALE, I suppose it might be SD), latent codes usually have just a few channels (like 3 or 4). In such case, I am not sure if AdaIN is capable of learning rather high-level style information (e.g. painting styles, instead of low-level style like colors).
3. It seems to me that the text prompts do not provide additional information as they are just describing what's in the images. Hence I am not sure if text prompts are really necessary for image composition.

### Minor issues and suggestions
4. In Sec. 4, $\odot$ and $\oplus$ are not clearly defined.
5. In Tab. 1--3 TALE and the baselines achieve scores that are very close for some of the metrics (e.g. LPIPS and content similarity). I am afraid that two digits after the decimal point are not sufficient in such cases and the authors should report more significant figures.
paper_summary: paper_summary: This paper presents TALE, a novel training-free framework that leverages the power of text-driven diffusion models to tackle the task of cross-domain image composition. TALE is a training-free method that operates directly on the latent space to provide explicit and effective guidance for the composition process. It is equipped with two mechanisms:
1. Adaptive Latent Manipulation: Formulates noisy latents conducive to initiating and steering the composition process by directly leveraging background and foreground latents at corresponding timesteps.
2. Energy-guided Latent Optimization: Exploits designated energy functions to further optimize intermediate latents to generate desired final results.
strength_weakness: strength_weakness: 1. Adequate evaluation: The paper conducts extensive experiments, comparing the proposed method with several state-of-the-art approaches across different domains. Quantitative metrics and qualitative results are provided to support the claims.
2. Applications: Suitable for more domain of images:1. This paper extend the image the baseline benchmark with more non-photorealistic samples and diverse backgrounds, and it outperforms on these non-photorealistic domain compared to existing methods.
limitations: limitations: 1. Lack of Novelty: The paper builds upon previous work on image composition, which is similar to TF-ICON , which have also explored the use of a training-free method through the manipulation of attention map in the diffusion models. The main improvement lies in the selection of inversion timesteps to employ
and it involves the AdaIn normalization, which is not the contribution of this work. It's not the first time to use the AdaIN normalization in style transfer [1] [2]. Besides, the energy-guided latent Optimization is quite naive and has none analysis why this is work for this task.
2. Insufficient Evaluation: The paper primarily focuses on presenting the technical details of TALE and its performance compared to prior methods. However, the evaluation could be strengthened by including.
3. Theoretical Approach: the section of energy-guided latent Optimization could benefit from more rigorous mathematical formulations and proofs to strengthen the theoretical arguments.
Chung, Jiwoo, Sangeek Hyun, and Jae-Pil Heo. "Style Injection in Diffusion: A Training- free Approach for Adapting Large-scale Diffusion Models for Style Transfer." arXiv preprint arXiv:2312.09008 (2023) (CVPR 2024).
paper_summary: paper_summary: This paper introduces TALE, a training-free framework using text-driven diffusion models for cross-domain image composition. TALE proposes two important tricks to improve TF-ICON.  First, it employs Adaptive Latent Manipulation to initiate and steer composition. Then, it presents Energy-guided Latent Optimization to refine intermediate latents. Their experiments show that TALE outperforms existing methods, achieving state-of-the-art results in both photorealistic and artistic domains.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow. 
2. Most experiments are solid and the results are fantastic. The ablation studies show how important the three components are.
3. The ideas of the Adaptive Latent Manipulation and Energy-guided Latent Optimization are simple but efficient. I am convinced that these two contributions help to improve the correctness of TF-ICON.
limitations: limitations: 1. The contributions are not so novel. The paper shows the importance of Selective 𝑻′, Normalization, and Energy-guided Latent Optimization in Image Composition. The ideas are not so novel and frequently used in diffusion models. I think a fixed $T$' and a fixed Optimization step $N$  may not be suitable for all images. Therefore, I recommend the authors improve the method by discussing how to adaptively choose $T$' and $N$ for each image instance.
2.  It seems that the ablation studies do not show the specific functions of each component. I am a little confused about the Figure 6. The authors say that Normalization is used for addressing significant domain discrepancies such as black-and-white $x_{bg}$ and colorful $x_{fg}$. However, in Figure 6, the result after adding Normalization is still colorful while the result after adding Optimization becomes black-and-white. I hope the authors give some explanation to the case.
3. Some small grammar issues exist in the paper. For example, $c$ in line 294 and $\mathbf{c}$ in line 297 are not consistent.