CFDiffusion: Controllable Foreground Relighting in Image Compositing via Diffusion Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:40:37

Abstract: Inserting foreground objects into specific background scenes and eliminating the gap between them is an important and challenging task. It typically involves multiple processing tasks, such as image harmonization and shadow generation, which find numerous applications across various fields including computer vision and augmented reality. In these two domains, there are already many mature solutions, but they often only focus on one of the tasks. Some image composition methods can address both of these issues simultaneously but cannot guarantee complete reconstruction of foreground content. In this work,we propose CFDiffusion, which can handle both image harmonization and shadow generation simultaneously. Additionally, we introduce a foreground content enhancement module based on the diffusion model to ensure the complete preservation of foreground content at the insertion location. The experimental results on the iHarmony4 dataset and our self-created IH-SG dataset demonstrate the superiority of our CFDiffusion approach.

Main paper: 
 introduction: 
 Image composition stands as a fundamental task in computer vision and augmented reality, aimed at seamlessly integrating objects from one image into another to craft a convincingly realistic composite image. Merely inserting the foreground object into a background image without careful consideration results in noticeable discrepancies between the foreground and background, such as differences in color, brightness, and shadows. Based on this, we can decompose the image compositing process into multiple subtasks, each addressing specific issues: image harmonization [2, 5, 6, 9, 19, 23, 48, 51] and shadow generation [17, 28, 44, 55] .Image harmonization aims to adjust the compatibility between the foreground and background in terms of color and brightness, while shadow generation ensures that the inserted foreground objects cast realistic and reasonable shadows. Various practical methods exist for both of these subtasks. However, employing multiple models to address each subproblem individually is both cumbersome and impractical. What we need is a unified network that can address both of these issues simultaneously, achieving excellent results for each. Figure 1 illustrates the problems we need to deal with and the results we should achieveIn recent years, generative models such as GANs [3, 18, 20, 43 ] and diffusion models [1, 16, 30, 34, 36, 39, 40] have demonstrated significant potential in image composition. Particularly, diffusion models have surpassed various preceding methods in image editing [1, 22, 34] and other applications [15, 33, 37] . Conditional diffusion model aims to generate images under the guidance of conditional information, such as text or semantic masks. Among them, Stable Diffusion (SD) [39] stands out as one of the most popular models, successfully integrating text from CLIP [38] into latent diffusion. Some existing works have introduced the diffusion model into related research domains, particularly in image editing and image composition. For example, SDEdit [34] composites images by adding noise to the input image and then iteratively denoises it through stochastic differential equations. However, these methods lacks proper and sufficient guidance during the denoising process, resulting in the final image lacking sufficient content fedelity. Besides, most diffusion models for image editing focus on manipulating images using text input, which is inappropriate for image composition.Recently, some image compositing methods [46, 52] have attempted to address all issues within a unified model, which can significantly simplify both model size and complexity. For example, ObjectStitch [46] utilizes a bounding box that encompasses the foreground object to specify the region for foreground object insertion and shadow generation, then processes the foreground within this designated area. Given the recent successful applications of diffusion models in image processing, these methods typically rely on pretrained diffusion models. However, in practice, these methods result in uncontrollable adjustments to the foreground in terms of both position and content texture, raising concerns about preserving the fidelity and credibility of the foreground object.In this paper, our aim is to address the issue of inadequate foreground fidelity observed in previous discussions on image composition. We introduce a method called CFDiffusion that concurrently handles image harmonization and shadow generation tasks. Building upon stable diffusion, we introduce a foreground content enhancement module (FCEM), which utilizes a foreground content encoder to extract foreground content information, thus guiding the reconstruction of foreground content. Furthermore, we equip SD with a lightweight adaptive encoder designed to extract crucial conditional information from the composite image, such as background style and color, to guide the denoising process of SD.To validate the effectiveness of our approach, we compare it with state-of-the-art methods and conduct experiments on benchmark datasets such as iHarmony4 [7] and our proposed IH-SG dataset. The experimental results demonstrate that our method achieves more realistic image harmonization and produces shadows that are both genuine and believable.Our contributions can be summarized as follows:• We introduce a novel image composition method called CFDiffusion. This method simultaneously handles image harmonization and shadow generation tasks for foreground objects with masked insertion points. • We design a foreground content enhancement module to fully reconstruct the foreground content and texture details. Extensive experiments conducted on both public datasets and our newly created dataset IH-SG validate the effectiveness of our proposed method. 
related work 2.1 image harmonization: 
 As a subtask of image compositing, the objective of image harmonization is to integrate objects from a given foreground image into a background image to create a cohesive composite image. This process involves adjusting the color and lighting information of the foreground object to ensure its compatibility with the background of the composite image.Traditional methods [26, 47] rely on adjusting the appearance of the foreground to match the color statistics of the background, typically focusing on obtaining color statistics and then transferring this information between the foreground and background. These methods are fast and straightforward but often struggle with complex scenes and produce artifacts because the realism of the image is often not well captured by these statistics.Particularly, with the release of the first large-scale image harmonization dataset, iHarmony4 [7] , supervised image harmonization methods [4, 8, 10, 11, 13, 27] have garnered increasing attention. For instance, [11] employed attention blocks to compute non-local information for foreground adjustment. SSAM [8] integrates them using a dual-path attention model, focusing on the relationship between spliced and unspliced regions. DoveNet [7] treats image harmonization as a domain translation task. CDT-Net [6] combines pixel-to-pixel and RGB-to-RGB transformations for high-resolution image harmonization. [27] introduced the concept of style from the background image, treating the harmonization task as a style transfer problem. They proposed a novel Region-Aware Instance Normalization (RAIN) method, which extracts style information solely from the background features and applies it to the foreground of the image harmonization task.However, when the task extends to shadow generation, these methods do not scale well to handle both tasks simultaneously.Previous work on shadow generation can be categorized into two main approaches: rendering-based methods and image-to-image translation methods.Rendering-based methods [21, 24] relies on a clear understanding of lighting, reflectance, material properties, and scene geometry to generate shadows for inserted virtual objects using rendering techniques. However, such knowledge is often unavailable or impractical for applications in real-world scenarios. Image-to-image translation methods predominantly employ deep learning techniques, characterized by encoder-decoder architectures. By training on paired images, including images with shadows and those without, these methods directly learn the mapping from shadow-free images to shadowed images from the input data. Importantly, this approach typically eliminates the need for explicit knowledge about lighting, reflectance, material properties, and scene geometry. The ARShadowGAN [28] model introduces an attention-guided network capable of directly modeling the mapping relationship between the shadows of foreground objects and their corresponding real environments, accompanied by the release of the Shadow-AR dataset. SGRNet [17] promotes comprehensive information interaction between the foreground and background. It initially predicts a mask for shadow regions and subsequently forecasts shadow parameters to fill these regions. Additionally, a new shadow training dataset, DESOBA, is introduced. ShadowGAN [55] combines both global conditional discriminators and local conditional discriminators to generate shadows for inserted 3D foreground objects without relying on background lighting information.Shadow generation is associated with the foreground objects, but it targets different areas than image harmonization. We aim to combine both aspects using a single network framework. 
proposed method: 
 Given a composite image 𝐼 𝑐 , a binary mask 𝑀 𝑏𝑠 representing the background object-shadow pair, and a binary mask 𝑀 𝑓 indicating the foreground object, our goal is to obtain an image 𝐼 that harmonizes the foreground object and produces reliable shadows under background illumination conditions.As illustrated in Figure 4 , our method consists of two stages: foreground object shadow mask prediction stage and shadow-harmonization generation stage. It mainly comprises four components: foreground object shadow generator 𝐺 𝑓 𝑠 , foreground content encoder 𝐸 (•), adaptive encoder 𝐸 𝑎 , foreground content enhancement module 𝐹𝐶𝐸𝑀, and shadow-harmonization generator 𝐺 (•) based on a stable diffusion model.The network workflow is as follows: Firstly, we use the background object-shadow data pair as reference to predict the shadow mask of the foreground object, identifying the approximate location for shadow generation. Then, we input the synthesized image into the shadow-harmonization generator to produce the final result. Simultaneously, we use the foreground content encoder to extract the foreground content embedding, inputting it into the foreground content enhancement module to constrain and complete foreground texture details. The adaptive encoder transfers the background style to the foreground region, providing additional generation guidance for the harmonization-shadow generator.Recently, diffusion models have shown remarkable performance in many fields: image generation [16, 45] , text-to-image generation [39] , image translation [25] , image inpainting [32, 41] , and image editing [12, 34] . The backbone of our Harmony-Shadow Generator is built upon a Stable Diffusion (SD) [39] model.SD is a latent diffusion model that undergoes a two-stage pretraining process, involving an autoencoder and a denoising U-Net [16] . In the first stage, the SD model trains an autoencoder: the encoder E converts the images I into a latent representation 𝑧 ′ 0 = E (𝐼 ), and then the decoder D reconstructs the images, resulting in ). In the second stage, the autoencoder's parameters are fixed, and SD introduces noise to the latent space representation 𝑧 ′ 0 over T steps to generate 𝑧 ′ 𝑡 . This process involves the creation of a denoising U-Net 𝜖 𝜃 , which is trained using a latent denoising lossEQUATIONHere 𝜖 is the noise added to the latent space feature 𝑧 ′ 0 at each noise step, 𝜖 𝜃 1 is the denoising U-Net that predicts the noise 𝜖 at the current step 𝑡, and y represents additional conditions (e.g. text, mask, etc.), 𝜏 𝜃 2 is instead a domain-specific encoder that projects y to an intermediate representation.In this work, we add conditional information using an adaptive encoder similar to that of composite images with foreground masks. During the inference process, noise is first added to 𝑧 ′ 0 to generate 𝑧 ′ 𝑇 , and then 𝑧 ′ 𝑇 is used as 𝑧 𝑇 , which is the initial input of 𝜖 𝜃 1 . Then iteratively use 𝜖 𝜃 1 to estimate the noise at each denoising step 𝑡, thereby gradually refining the latent map 𝑧 𝑇 , and ultimately become a clean latent feature 𝑧 0 . Finally, the clean latent features 𝑧 0 are fed to the decoder D to generate images. Inspired by [17] , we apply a shadow mask predictor to generate foreground object shadow predictor. First, we predict the foreground shadow mask 𝑀 𝑓 𝑠 through the foreground shadow mask generator 𝐺 𝑠 . 𝐺 𝑠 consists of an encoder and a decoder 𝐷, and the encoder is divided into a foreground encoder 𝐸 𝑓 and a background encoder 𝐸 𝑏 . We believe that the background object-shadow pair contains clues that are beneficial for inferring the foreground shadow area.In order to generate the shadow mask of the foreground object, we take the concatenation of the composite image 𝐼 𝑐 and the background object-shadow mask 𝑀 𝑏𝑠 as the input of the background encoder 𝐸 𝑏 , and generate the background feature map 𝑋 𝑏 . At the same time, the concatenation of the composite image 𝐼 𝑐 and the foreground object mask 𝑀 𝑓 is used as the input of the foreground encoder 𝐸 𝑓 to obtain the foreground feature map 𝑋 𝑓 . The process is summarized as follows:EQUATIONEQUATIONFollowing [17] , we use a Cross-Attention Integration (CAI) [17, 49, 50, 54] layer to help the foreground feature map notice the relevant lighting information of the background feature map. As the picture 3 shows, the input of the CAI layer consists of 𝑋 𝑓 and 𝑋 𝑏 , which are outputs of foreground encoder 𝐸 𝑓 and a background encoder 𝐸 𝑏 , and the output feature map is denoted as 𝑋 . Then 𝑋 is fed into the decoder 𝐷 to obtain the mask of the foreground object shadow. Subsequently, we add it to the foreground object mask 𝑀 𝑓 to obtain the foreground object-shadow mask 𝑀 𝑓 𝑠 , which serves as one of the inputs for the subsequent shadow-harmonization generator. The process is summarized as follows:EQUATION)Following [53] , in order to further enhance the detailed texture of the foreground generation, we employ the pre-trained model 𝑉 𝑖𝑇 -𝐿/14 from CLIP [38] as the foreground image encoder 𝐸. Initially, we extract the foreground object region from the synthesized image 𝐼 𝑐 using the foreground object mask 𝑀 𝑓 ,which is then inputted into the foreground image encoder 𝐸 to extract the local content embedding of the foreground 𝐸 𝑙 . This process can be represented as follows:EQUATIONThe intermediate layer of the CLIP encoder outputs 256 patch tokens containing local details. We extract the information of these patch tokens and integrate these foreground content embeddings into the Foreground content enhancement module (FCEM) of the denoising U-Net model to help us control the generation of foreground content details. The specific details of the FCEM module is located in Section 3.4.Following [53] , we utilize a foreground content enhancement module to embed foreground content into the intermediate features of the diffusion model, thereby constraining the stable diffusion model for foreground appearance generation and promoting the composite generation of foreground appearance with high fidelity.Our foreground content enhancement module is built upon the publicly released v1-4 SD model. To identify foreground regions that need to be constrained, we append the binary foreground object-shadow mask 𝑀 𝑓 𝑠 to the model input. To achieve this, in the first convolutional layer of U-Net, we attach two additional input channels to respectively contain the foreground object mask 𝑀 𝑓 𝑠 Denoising U-Net of SD consists of a series of basic blocks, each block includes a residual block and a transformer block. The transformer block consists of a self-attention module, a cross-attention module, and a feedforward network.As illustrated in Figure 5 . We record the features from the 𝑖 -𝑡ℎ transformer block as 𝐹 𝑖 ∈ R ℎ 𝑖 ×𝑤 𝑖 ×𝑐 𝑖 , where ℎ 𝑖 , 𝑤 𝑖 , 𝑐 𝑖 represent its height, width, and channel dimensions respectively. We first use the foreground local feature 𝐹 𝑙 𝑖 intercepted by the foreground object mask (𝑀 𝑓 ) resized to ℎ 𝑖 , 𝑤 𝑖 , 𝑐 𝑖 . The feature map will be flattened to F𝑙 𝑖 ∈ R 𝑁 ×𝑐 𝑖 , and then passed through cross attention together with the foreground local embedding 𝐸 𝑙 , and then we get an attention map 𝐴 and refined foreground local feature map 𝐹 𝑙 𝑖 . Then align the foreground local embedding 𝐸 𝑙 with 𝐹 𝑙 𝑖 to obtain the aligned foreground embedding map 𝐸 𝑙 . Further use 𝐸 𝑙 to modulate 𝐹 𝑙 𝑖 . 𝐸 𝑙 is passed through a convolutional layer of 3 × 3 to obtain the spatial awareness modulation weight, and the modulation is normalized The transformed 𝐹 𝑙 𝑖 is as follows:EQUATIONFinally, after resizing F𝑙 𝑖 , it is added to the foreground object region of 𝐹 𝑖 . The output of the Foreground Content Enhancement Module (FCEM) is then delivered as the enhanced foreground content features 𝐹 𝑖 to the next residual block.Following [31, 35] , we adopt an adaptive encoder, which is a lightweight model that can align the internal knowledge in the SD model with external control signals. Through this adaptive encoder, we can achieve rich control effects on the color and structure of the SD generation results.The adaptive encoder takes into account encoding additional conditions and provides multi-step guidance for denoising U-Net in the denoising step. Previous adaptive encoder implementations focused more on coarse structures (e.g., sketches, poses, semantic masks) and exploited textual conditions to indicate additional requirements (e.g., style or context). Different from previous work, we abandon the text CLIP model, splice the composite image 𝐼 𝑐 and the foreground mask 𝑀 𝑓 , and use a lightweight adaptive encoder to encode while retaining content details and extracting background styles. The structure of the adaptive encoder includes four feature extraction blocks and three DownSample (DS) blocks.First, the input image will be resized to 64×64, and we name it 𝐹 0 𝑐 . Each feature extraction module (EM) includes a convolutional layer and two residual blocks. The generation process of 𝐹 𝑖 𝑐 , 𝑖 ∈ 1, 2, 3, 4 can be expressed as follows:EQUATION)EQUATION)The resolutions of 𝐹 EQUATIONEQUATIONAmong them, 𝑀 𝑓 represents the foreground object mask scaled to the corresponding 𝐹 𝑖 𝑐 size, • represents element-wise product, and 𝐹𝑙𝑎𝑡𝑡𝑒𝑛(•) represents expanding a 2D feature map into a 1D feature feature sequence.We use a transformer layer to extract and transfer the style of the background to the foreground area to achieve harmonious processing of the foreground. In addition, the parts of the background area that are related to the foreground can provide more references when harmonizing the foreground, so they are very important. We will also pay more attention to the areas that are more related to the background and foreground. 𝐹 𝑖 𝑐,𝑓 is used as query, 𝐹 𝑖 𝑐,𝑏 is used as keys/values, and the final background stylized foreground feature F𝑖 𝑐,𝑓 can be expressed as: F𝑖 𝑐,𝑓 = 𝑇𝑟𝑎𝑛𝑠 𝑓 𝑜𝑟𝑚𝑒𝑟 (𝐹 𝑖 𝑐,𝑓 , 𝐹 𝑖 𝑐,𝑏 , 𝐹 𝑖 𝑐,𝑏 ). (11)Our total loss function 𝐿 𝑡𝑜𝑡𝑎𝑙 consists of the standard noise loss L 𝐿𝐷𝑀 of the diffusion model and a reconstruction loss 𝐿 𝑟𝑒𝑐 , Therefore, the final loss function of our CFDiffusion is:EQUATIONwhere 𝜆 1 , 𝜆 2 ,𝜆 3 are hyper-parameters which control the influence of terms. Noise Loss. First, we adopt the standard noise loss of the diffusion model, aiming to reconstruct the image features in the latent space, shown as Equation ( 13):EQUATION)Reconstruction loss. It is a classical 𝐿 1 loss between the generator output image Î and real ground-truth image 𝐼 , to further constrainInput SGDiffusion ARShadowGan ObjectStitch DucoNet DIH-GAN Ours GTFigure 6 : The comparison between our method and two image harmonization methods: DucoNet [48] and DIH-GAN [2] , three shadow generation methods [28, 29, 46] on our dataset. It can be clearly seen that our CFDiffusion has achieved the best results in both real shadow generation and image harmonization.the generated image towards the ground truth, which is expressed as:EQUATION) MSE Loss. Additionally, we compute the loss for the foreground mask prediction module using the following method:EQUATION) 
experiments: 
 To verify the superiority of our proposed CFDiffusion, we compare CFDiffusion with state-of-the-arts on the real-world iHarmony4 [7] dataset and our proposed dataset, and provide assessments both quantitatively and qualitatively.The proposed method is implemented using PyTorch, which is trained using one NVIDIA RTX 3090 GPU. All images are resized to 256 × 256 for training and testing. We adopt adam optimizer with the momentum as (0.9,0.999), and the learning rate initialized as 0.00003. Following [42] , we use the Kaiming initialization technique [14] to initialize the weights of the proposed model and use a 0.9999 Exponetial Moving Average(EMA) for all our experiments. We used 1000 diffusion steps T and noise schedule 𝛽 𝑡 linearly increasing from 0.0001 to 0.002 for training, and 25 steps for inference. After a few trials, we set 𝜆 2 = 𝜆 3 = 10, 𝜆 1 = 1 by observing the grenerated images. The training epoch is set as 1500.Compared methods. We compared our model with five deep learning-based methods from the related fields: two image harmonization methods including DucoNet [48] , DIH-GAN [2] , three shadow generation methods include ARShadowGAN [28] , SGDiffusion [29] , and ObjectStitch [46] . Among them, SGDiffusion is the latest method using a diffusion model for shadow generation tasks, while ObjectStitch is the newest image composition method that addresses both image harmonization and shadow generation tasks. DucoNet for image harmonization based on dual color spaces. ARShadowGAN makes full use of the background information to guide the shadow generation of foreground objects. We train and test all these methods based on our dataset. For detailed information about our IH-SG dataset, please refer to the remainder of this section.In addition, we also tested our image harmonization capabilities on the iHarmony4 dataset, and compared the three image harmonization methods of CDT-Net [6] , Harmonizer [23] and DucoNet [48] to further prove the superiority of our CFDiffusion. CDT-Net coherently combines pixel-to-pixel conversion and RGB-to-RGB conversion in an end-to-end network.Evaluation metrics. We use four metrics to evaluate the image illumination harmonization results, which are Relative Mean Square Error (RMSE), Structural Similarity Index Measure (SSIM), foreground Mean Square Error (fMSE), foreground Structural Similarity Index Measure (fSSIM). Generally, the smaller RMSE and fMSE, and the larger fSSIM and SSIM indicate the better image illumination harmonization results.Dataset. To better train our model, we have constructed a dataset called IH-SG that can address both of image harmonization and shadow generation concurrently. Each data pair we construct includes: composite images, ground truth images, foreground object masks, foreground object shadow masks, background object masks and background object shadow masks.In total, our dataset comprises over 1000 outdoor scenes and more than 10000 data pairs. We also captured numerous shadow scenes under complex conditions, such as shadows cast on walls and steps, to enrich the shadow data samples, making our dataset more realistic and diverse. Our data pairs are illustrated in the Figure 2 . Experiments on our dataset. The quantitative comparison results of different methods on our testing set are summarized in Table 1 . Apparently, our method CFDiffusion achieves the better quantitative results than other state-of-the-art methods on all four metrics. On the one hand, this is due to the powerful image generation capability of the diffusion model. On the other hand, our FCEM and adaptive encoder make full use of the foreground and background information, proving the superiority of our CFDiffusion.Experiments on iHarmony4. The iHarmony4 dataset is one of the most popular large-scale datasets in the field of image harmonization, covering a variety of scenes and foreground objects. Therefore, we compared our method with several image harmonization techniques on the iHarmony4 dataset, and the quantitative comparison results are shown in Table 3 . Our proposed approach also achieved the best results.Experiments on DESOBAv2. Recently, Liu et al. [29] extended the DESOBA [17] dataset to DESOBAv2, which has become the latest shadow generation dataset. To validate the generalization ability of our model, we conducted experiments on the DESOBAv2 dataset by applying slight perturbations to foreground objects, and the results are shown in Figure 13 .In order to verify the effectiveness of each component of our method, we conduct ablation studies by modifying the CFDiffusion architecture. Specifically, we set the following variants:To verify the crucial roles played by our FCEM module and adaptive encoder module in the overall model, we set up several variants. Firstly, we chose the original diffusion model as the baseline, which is referred to as "baseline" in Table 2 . To demonstrate the pivotal roles of the FCEM module and adaptive encoder module in the entire model, we removed these two modules separately from the complete model, which are referred to as "w/o FCEM" and "w/o adaptive encoder" in Table 2 . Finally, we compared these variants with the full model "Ours (full model)" for comprehensive analysis, and some results are shown in Figure 12 .Result GT Figure 10 : In order to simulate scenarios where there are no background objects available as reference, we only utilize the foreground encoder module during the foreground shadow mask prediction stage, hence not referring to background shadow information. Consequently, the inferred shadow shapes will deviate significantly from the ground truth. We trained these variants using the same training data and quantitatively evaluated their impact on the test results. The evaluation results are presented in Table 2 . From the table, we can observe that: after introducing guided supervision with FCEM, the model's quantitative performance has made significant strides, sufficiently demonstrating the strong guiding role of the FCEM module in capturing texture details of foreground objects. Moreover, with the inclusion of the adaptive encoder, the model's performance has also noticeably improved compared to the original diffusion model, confirming its guiding role in the harmonization generation of foreground objects.With the simultaneous introduction of both the FCEM and adaptive encoder modules, our full model achieved the best performance, demonstrating the effectiveness of our approach. Additionally, incorporating the FCEM and adaptive encoder modules into the original diffusion model significantly improves performance. Our CFDiffusion still has the following limitations: (1) As shown in Figure 10 , for scenarios lacking background object references or involving complex situations where shadows are cast onto intersecting planes, our model struggles to generate shadows effectively.(2) Due to computational costs and processing speed limitations, our method is currently not applicable to real-time video lighting harmonization, which is also one of our future directions for improvement.In this paper, we proposed an image composite method based on the diffusion model, which focuses on harmonizing the illumination inconsistency between foreground objects and background, while generating realistic shadows. We employed an adaptive encoder to extract style features from the background to guide the experiments on our proposed IH-SG dataset, as well as the popular DESOBAv2 dataset and iHarmony4 dataset, demonstrating that our method achieves significant improvements. In the future, we will expand CFDiffusion to adapt to real-time video lighting harmonization and shadow generation.  
 
 Appendix:  


Decision: This paper receives a mixture of reviews, two borderline accepts, one weak reject, and one reject. The reviewers giving borderline accepts acknowledge the motivation and effectiveness of the proposed method. The review of the reviewer giving reject is very short and not very informative. The concern of the other reviewer giving weak reject mainly lies in the novelty of the proposed method, which seems to be a combination of existing methods. I believe that the constructed dataset is a huge contribution to the community. The proposed method is also not a naive combination and has some interesting designs. The first stage provides useful prior information for the second stage, which alleviates the burden of diffusion model in the second stage. The foreground information is injected into denoising UNet through two ways to ensure the foreground quality. By weighing the advantages and drawbacks, I tend to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes CFDiffusion to simultaneously handle image harmonization and shadow generation. The experimental results on the iHarmony4 dataset and our IH-SG dataset demonstrate the superiority of CFDiffusion.
strength_weakness: strength_weakness: The results seems good.
limitations: limitations: 1.	The proposed method is a mix of existing methods or models (SGRNet, diffusion model) and lacks novelty. 
2.	The authors frequently mention the IH-SG dataset as their constructed dataset. For example, line 49 (our IH-SG), line 153 (our proposed IH-SG dataset), line 165 (our newly created dataset IH-SG). If the IH-SG dataset is the contribution, please add it to the paper. If the dataset is publicly available, I'm considering whether this breaks the principle of anonymity.
3.	The manuscript feels like being rushed. In the abstract, two datasets are claimed (line 49). In the conclusion section, three datasets are claimed (Line 923-924).
paper_summary: paper_summary: This paper proposes CFDiffusion, a unified model that utilizes diffusion models to simultaneously tackle image harmonization and shadow generation tasks.  It includes two main stages: foreground object shadow mask prediction stage and shadow harmonization generation stage. This method has been evaluated with comprehensive experiments.
strength_weakness: strength_weakness: This paper is well-organized and straightforward. The experiments clearly demonstrate the advantages of the proposed pipeline, particularly showing the benefits of the adaptive encoder and the Foreground Content Enhancement Module (FCEM).
limitations: limitations: [1]. Please show the inference time between the different methods.

[2]. Are there any quantitative results comparisons on the DESOBAv2 dataset?

[3]. You can add more details about FCEM in part 3.4 to illustrate the Figure5.
paper_summary: paper_summary: The paper presents CFDiffusion, a unified model simultaneously handling image harmonization and shadow generation. After first employing a mask predictor to obtain a shadow mask, the paper proposes to use FCEM and an adaptive encoder to help with appearance preservation and harmonization.
strength_weakness: strength_weakness: - The motivation of combining shadow synthesis and harmonization is reasonable, as these effects share some common information from the global background area.
- The paper shows extensive quantitative and visual results, which demonstrate the advantages of the proposed model.
limitations: limitations: - Fig 2 is not mentioned until Sec 4.1. It would be better to switch the order of the figures.
- Fig 4 needs to be revised for easier understanding. It could be improved in the following ways: 1) some notations should be added to the figure, such as G_fs; 2) maybe use two boxes to illustrate the two stages; 3) the exact inputs should be visualized to show how they are processed by the encoders, e.g., what’s the input of E_a? Is it the masked composite image or the concatenation of I_c and M_f?
- Fig 5 is confusing. How is the “align” operation performed?
- An ablation on the reconstruction loss is missing. This would be beneficial to show whether identity preservation is contributed by FCEM or the reconstruction loss.
- (Minor) For the comparison with baselines, maybe also consider AnyDoor and ControlCom (ControlCom has a harmonization mode).
- How does the model perform in indoor scenes where there are soft shadows? Such examples are missing from the paper.
- In the third row of Fig 11, it seems the proposed model does not have an obvious advantage over the baselines?
- In Tab 3, DucoNet should be the one highlighted regarding the fMSE score.
paper_summary: paper_summary: This article presents a decomposition of the image synthesis task into two subtasks: shadow generation and image harmonization. Subsequently, the article proposes an image composite method based on the diffusion model. This method seamlessly integrates foreground objects into a given background scene and relights foreground objects while generating shadows to better blend them into the background and eliminate any sense of incongruity between them.
strength_weakness: strength_weakness: 1.The experimental results indicate that the proposed approach can effectively address the issue of insufficient foreground fidelity in image synthesis.

2.This work constructs a dataset that can address both of image harmonization and shadow generation concurrently.
limitations: limitations: About the method
The article's primary deficiency is the lack of innovation. It appears that there is no difference between the foreground shadow mask generator module in this paper and the shadow mask prediction module in [1], and the foreground encoder module and FCEM in this paper are no different from the module of the same name in [2]. The whole paper is basically composed by adding an auto-encoder module on the basis of [1] and [2].

About the experiments
The work in this article contains two subtasks, image coordination and shadow generation, but the experimental part of it is not particularly well described.

1.SGDiffusion is just a shadow generation method, whereas Table 1 represents image coordination work and how you use it to perform image coordination work on your dataset. Similarly, the other methods compared are either shadow generation or image coordination single-task solutions, and the article lacks content on how that component is a fair comparison。

2.The DESOBAv2 dataset is a shadow generation dataset. Are the experiments on this dataset simply to assess the ability of shadow generation? Additionally, the article lacks objective metric results on that dataset. Would the results be consistent with those of [1]? (How does the article's approach to shadow generation differ from [1]?)

3.The caption of Figure 12 refers to five images, however, there are six images within the picture. It is therefore necessary to re-add the description.

About the writing

1.Row 506 seems to be missing a full stop.

2.Why does the formula 11 input have two identical elements？
The author should carefully review their manuscript for further writing errors.

[1]Yan Hong, Li Niu, and Jianfu Zhang. 2022. Shadow Generation for Composite Image in Real-World Scenes.
[2]Bo Zhang, Yuxuan Duan, Jun Lan, Yan Hong, Huijia Zhu, Weiqiang Wang, and Li Niu. 2023. Controlcom: Controllable image composition using diffusion model.