Foreground Harmonization and Shadow Generation for  Composite Image

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:24:55

Abstract: We propose a method for light and shadow editing of outdoor disharmonious composite images, including foreground harmonization and cast shadow generation. Most existing work can only perform foreground appearance editing tasks or only focus on shadow generation. In fact, lighting not only affects the brightness and color of objects, but also produces corresponding cast shadows. In recent years, diffusion models have demonstrated their strong generative capabilities, and due to their iterative denoising properties, they have a significant advantage in image restoration tasks. But it fails to preserve content structure of image. In this purpose, we propose an effective model to tackle the problem of foreground light-shadow editing. Specifically, we use a coarse shadow prediction module (SP) to generate coarse shadows for foreground objects. Then, we use the predicted results as prior knowledge to guide the generation of harmony diffusion model. In this process, the primary task is to learn lighting variation to harmonize foreground regions. The secondary task is to generate high-quality cast shadow containing more details. Considering that existing datasets do not support the dual tasks of image harmonization and shadow generation, we construct a real outdoor dataset, IH-SG, covering various lighting conditions. Extensive experiments conducted on existing benchmark datasets and the IH-SG dataset demonstrate the superiority of our method.

Main paper: 
 introduction: 
 Image coposite refers to the process of combining images from different sources to create new images, which has a variety of applications like advertisement propaganda and digital entertainment. However, due to variations in lighting conditions, camera parameters, and other factors, composite images often have inconsistent lighting statistics compared to real images. This necessitates image harmonization to adjust the appearance of the foreground for visual consistency. Additionally, most existing image harmonization methods focus solely on the lighting effects of foreground regions. However, lighting also produces corresponding shadow effects. These shadows provide important clues about object shape, position, and relative depth, conveying information about volume and depth to observers. Therefore, shadow generation is equally essential for achieving lighting-shadow consistency.For image harmonization, most traditional methods focus on better matching low-level appearance statistics. They mainly focus on color correction and lighting compensation [3, 31, 33, 49] for the foreground area, or adjustments based on gradient information [18, 30, 45] and multiscale statistics [43] of foreground and background regions. They fail to solve the significant appearance differences between foreground and background images. Deep learning based methods provide powerful capabilities for modeling regional appearances to facilitate harmonization. Some methods [7, 47] explore the semantic information of synthesized images to reconstruct coordinated images. Several methods [4, 6, 14, 23] have been proposed to explore domain adaptation, aiming to bring the predicted foreground closer to the original background domain. Guo et al. [13] introduced the Retinex theory into the image harmonization task. These methods [11, 12] explored Transformer-based image harmonization. Considering the limitations of existing datasets, selfsupervised or semi-supervised methods proposed [19, 27, 29, 48] . CDTNet [5] and PCT-Net [9] adjust the image through color transformation, but ignore lighting effects. Tan et al. [44] used unrelated L, a, b features to guide image reconstruction.For shadow generation, render-based methods [39] [40] [41] require explicit knowledge of lighting, reflectance, material properties, and scene geometry to generate shadows for inserted virtual objects using rendering techniques. However, obtaining such knowledge is often impractical in real-world scenarios. The estimated results are influenced by the accuracy of the input information [8, 17] . Deep learing-based methods [16, 25] , on the other hand, directly learn the mapping from input images without foreground shadows to output images with foreground shadows, without the need for explicit knowledge of lighting, reflectance, etc. Bao et al. [1] consider the harmonized appearance and illumination of the foreground objects while generating reasonable shadows for the foreground objects. But this method only studied indoor images.To address these issues, we propose a novel method for both image harmonization and shadow generation in this paper. Considering the powerful generation capability of diffusion models, inspired by [10] , we use a condition diffusion model as the backbone network. Recognizing that images provide rich structural and semantic features to assist image reconstruction compared to textual information. We use composite image with coarse shadows as condition to guide the diffusion model. Since our task also requires generating plausible cast shadows for the foreground objects, we introduce a feature extraction module, a channel-spatial cross attention and a shadow generation model to generate coarse shadows. The main task of the harmony diffusion model is to learn the lighting changes in the background to handle the foreground area harmoniously, and refine the shadow area to generate high-quality shadows.The existing dataset are not well-suited for our task. IHarmony4 [6] provides different color conversions but lacks attention to lighting. RealHM [19] and RdHarmony [2] require a significant amount of manpower and technical resources. CcHarmony [29] focuses on realistic lighting changes, but it requires high requirements for the shooting process. ShadowAR [25] dataset is collected through rendering models. However, the attributes of shadows may also not match those of real images. DESOBA and DESOBAv2 [16, 26] use real images as target images to remove shadows from the foreground to generate composite images. Bao et al. [1] proposed an indoor dataset for foreground harmonization and shadow generation, but only focusing on indoor scenes. So we construct a new outdoor real-world dataset (IH-SG) for image harmonization and shadow generation task. We obtain composite images through relighting foreground objects.Our contributions can be summarized as follows:• We constructe a new outdoor real-world dataset (IH-SG) for image harmonization and shadow generation task. • We propose a new image light-shadow editing method based on condition diffusion model, which can achieve controllable harmonization of foreground regions and reasonable generation of cast shadows.Extensive experiments conducted on both public datasets and our IH-SG dataset demonstrate the effectiveness of our method. 
related work: 
 Our task aims to handle the illumination of the foreground regions and generate reasonable cast shadows for foreground objects, while image harmonization or shadow generation tasks can only solve one of them.Traditional image harmonization methods primarily focus on adjusting the low-level appearance statistics between foreground objects and the background, such as color statistics [3, 31, 33, 49] , and gradient information [18, 30, 45] . The limited representation capability of low-level features can negatively impact their performance. Especially when there are significant differences between the foreground and background regions, these methods often struggle to produce satisfactory results.Recent research has built reasonably sized datasets [6, 19, 29] , to advance learning-based approaches. CNN-based methods analyze semantic information [7, 47] . Since image harmonization adjusts the foreground lighting or style to match the background, domain adaptation methods [4, 6, 23] have also been proposed to explore the idea of domain harmonization. Guo et al. [13] introduced the Retinex theory into the image harmonization task and proposed decomposing the synthetic image into reflectance and illumination. With the rise of Transformers, Guo et al. [11, 12] applied the Transformer framework to image harmonization tasks. But intrinsic decomposition is a difficult problem. Some methods treat image harmonization as a style transfer problem. These methods have achieved advanced research results through contrastive learning [14] , high resolution [20] or color space adjustment [5, 9, 44] . Shen et al. [38] trained Global Perception Adaptive Coordination Kernel. Bao et al. [1] utilized a multi-scale attention mechanism and illumination exchange strategy to harmonize objects and generate cast shadow. However, these methods for indoor scenes are difficult to generalize to outdoor scenes. Unlike existing methods, we learn the illumination of images through diffusion model to generate the consistent illumination of the foreground object itself with the background, as well as corresponding cast shadow.The existing work on shadow generation can be divided into two categories: rendering-based methods and image-to-image translation methods. Rendering-based methods require explicit knowledge of lighting, reflectance, and scene geometry to generate shadows for inserted virtual objects using rendering techniques. However, such detailed knowledge relies on user input [21, 24] or model prediction [8, 22] . Sheng et al. [40] explored the generation of controllable soft shadows. And then, they introduced the concept of pixel height [39, 41] and explored the correlation between objects, ground, and camera poses. In the absence of user interaction, Gardner [8] attempted to recover explicit lighting conditions and scene geometry based on a single image, but inaccurate estimates may lead to unsatisfactory results. Image-to-image translation methods learn the mapping from input images without foreground shadows to output images with foreground shadows, without requiring explicit knowledge of lighting, reflectance, etc. Hu et al. [17] proposed a method that can adapt to different scenarios, but failed to generate shadows in complex scenes. For instance, ShadowGAN [52] utilized both global and local conditional discriminator to enhance the realism of generated shadows. Liu et al. [25] released the ShadowAR dataset and proposed an attention-guided network for shadow generation. Yan et al. [16] addressed real-world scenes and generated plausible shadows. SGDiffusion [26] focused on the shadow generation problem based on a diffusion model. DMASNet [46] decomposed shadow mask prediction into box prediction and shape prediction. However, the shadows generated by these methods are still not accurate enough. .Diffusion-based generative models recently produced amazing results with improvements adopted in denoising diffusion probabilistic models [15] , which becomes increasingly influential in the field of low-level vision tasks, such as superresolution [37] , inpainting [28] , and colorization [36] . The methods [32, 34, 35] explored different modal conditions into the diffusion process, achieving controllability of the generated content of the generative model. Pallette [36] was proposed as a general image-to-image framework to solve the image restoration with conditional denoising diffusion probability models. The methods [50, 51] were proposed to generate results towards expectations. However, most of these methods focus on synthetic degradation, such as image coloring, image restoration, and super-resolution. In this article, we explore the problem of foreground harmony and shadow generation in the real world with limited training pairs. We build our model upon shadowdiffusion [10] to address the above issues. 
ih-sg dataset: 
 Lighting not only results in different color brightnesses of foreground objects but also leads to the generation of corresponding cast shadows. In this article, we simultaneously focus on the harmony of foreground objects and the generation of corresponding realistic shadows. Therefore, we have constructed a high-quality real outdoor dataset IH-SG, including composite images 𝐼 𝑐 , real images 𝐼 𝑟𝑒𝑎𝑙 , foreground obeject masks 𝑀 𝑓 𝑜 , foreground shadow masks 𝑀 𝑓 𝑠 , background object masks 𝑀 𝑏𝑜 and background shadow masks 𝑀 𝑏𝑠 .We take photos outdoors that meet our requirements, including background images, real images, and relighting images: Background images 𝐼 𝑏𝑎𝑐𝑘 : Choose optimal weather conditions and time periods with moderate lighting, avoiding excessively dim or harsh lighting conditions. Additionally, steer clear of rainy days or periods around sunrise or sunset, as these times exhibit significant changes in lighting. Select appropriate shooting angles and positions, stabilize the camera on a tripod, and control it via a mobile device to ensure camera stability.Real images 𝐼 𝑟𝑒𝑎𝑙 : Without altering camera parameters, positions, etc., placing foreground objects and capturing images to be used as real images in the training set. The camera remains stable throughout the process, ensuring consistency between background and real images, thus reducing alignment and correction efforts during subsequent image synthesis. Rapid placement of foreground objects generally assumes minimal changes in lighting between background and real images captured in a short time.Relighting foreground images 𝐼 𝑖 𝑟𝑒𝑙𝑖𝑔ℎ𝑡 : Without altering the shooting scene or camera position, we employ appropriately sized and shaped shading equipment to shade the scene, ensuring that neither the camera nor the objects are affected. Shading equipment is utilized to effectively block external light interference. Placing lights and adjusting their brightness and direction to illuminate foreground objects. Camera parameters such as exposure time, aperture size, and ISO sensitivity are adjusted based on the actual shooting environment and lighting conditions to achieve the desired exposure effects. Subsequently, adjust the lighting conditions to capture different relighted images 𝐼 𝑖 𝑟𝑒𝑙𝑖𝑔ℎ𝑡 , 𝑖 ∈ 𝑁 , where N represents different lighting conditions.Based on the acquired real images 𝐼 𝑟𝑒𝑎𝑙 , background images 𝐼 𝑏𝑎𝑐𝑘 , and relighting images 𝐼 𝑖 𝑟𝑒𝑙𝑖𝑔ℎ𝑡 , composite images 𝐼 𝑐 are obtained. To obtain refined data, we used Photoshop to obtain corresponding masks, including foreground object masks 𝑀 𝑓 𝑜 , foreground shadow masks 𝑀 𝑓 𝑠 , background object masks 𝑀 𝑏𝑜 , and background shadow masks 𝑀 𝑏𝑠 . Then, we use the obtained foreground object masks to extract relighting foreground objects and merge them with background images to generate composite images:EQUATIONThen, 𝐼 𝑐 and 𝐼 𝑟𝑒𝑎𝑙 form a pair of input composite image and groundtruth target image. Due to shooting conditions, there may be significant differences between the background images and the real images in the background area. If there are differences in color or brightness, they could be adjusted using style transfer techniques. Additionally, some unsuitable images could be filtered out. After that, we obtained tuples in the form of 𝐼 𝑐 , 𝑀 𝑓 𝑜 , 𝑀 𝑓 𝑠 , 𝑀 𝑏𝑜 , 𝑀 𝑏𝑠 , 𝐼 𝑟𝑒𝑎𝑙 , which will be used for model training. 
method 4.1 problem definition: 
 The input is a tuple (𝐼 𝑐 , 𝑀 𝑓 𝑜 ), where 𝐼 𝑐 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 , with 𝐻 and 𝑊 representing the height and width of image, and 𝑀 𝑓 𝑜 ∈ 𝑅 𝐻 ×𝑊 ×1 . This model aims to generate foreground object that is consistent with the background, and to generate reasonable cast shadow.The coarse shadow prediction module aims to predict cast shadows for foreground objects, including a background feature extraction network (BE) and a shadow generation network (SG). The composite image and foreground object mask are the inputs.For the shadow generation task, while complete background information may provide more details, it does not directly yield reasonable shadows for image-to-image transformation networks. This is because it may not adequately focus on crucial areas in the background, such as objects and their shadow information. Therefore, the BE module can learn relevant information from the background image to generate attention maps for reference objects and their shadows.The module adopts an encoder-decoder network with an attention mechanism as basic architecture, comprising an encoder and two decoders. The composite image without foreground object shadow andforeground object mask are concatenated along the channel dimension and serve as input to the encoder 𝐸. The extracted high-level features are fed into two separate branches of decoders. One decoder 𝐷 1 predicts the reference object mask 𝑀 𝑏𝑜 , while the other decoder 𝐷 2 predicts the corresponding shadow mask 𝑀 𝑏𝑠 :EQUATION𝑀 𝑏𝑠 = 𝐷 2 (𝐸 (𝐼 𝑐 )).(3)Module. Given a composite image 𝐼 𝑐 without foreground shadow and a foreground object mask 𝑀 𝑓 𝑜 , this module aims to generate coarse foreground shadow 𝐼 𝑠ℎ𝑎𝑑𝑜𝑤 . The specific network structure is as follows: Two same encoders, one decoder, and one special channel-spatial cross-attention mechanism (CSCA). Through the BE module, we can identify key areas in the background image that are beneficial for shadow generation. Inspired by [16] , in order to better utilize the information in the background, we adopt foreground encoder 𝐸 𝐹 and background encoder 𝐸 𝐵 , respectively. The foreground encoder 𝐸 𝐹 takes the concatenation of the composite image 𝐼 𝑐 and the foreground object mask 𝑀 𝑓 𝑜 as input, generating a foreground feature map 𝑋 𝑓 . The background encoder 𝐸 𝐵 takes the concatenation of 𝐼 𝑐 and 𝑀 𝑏𝑜𝑠 as input to generate a background feature map 𝑥 𝑏 :EQUATIONEQUATIONTaking inspiration from existing attention methods, we introduce a channel-spatial cross-attention (CSCA) to assist the foreground feature map 𝑋 𝑓 in obtaining relevant reference information from the background feature map 𝑋 𝑏 . Then, the decoder D is used to predict coarse shadow images for foreground objects: .EQUATIONto 𝑋 𝑟 𝑓 ∈ 𝑅 𝑊 𝐻 ×𝐶 and 𝑋 𝑏 ∈ 𝑅 𝑊 ×𝐻 ×𝐶 to 𝑋 𝑟 𝑏 ∈ 𝑅 𝑊 𝐻 ×𝐶 . Then, we compute the dependencies between any two elements of 𝑋 𝑓 and 𝑋 𝑏 in the global context:EQUATIONUsing the obtained similarity map A, we incorporate information from 𝑋 𝑟 𝑓 , then reshape it, and obtain the weighted feature map 𝑋 𝑏2 :EQUATION)Spatial cross-attention: Similar to the channel cross-attention, we reshape 𝑋 𝑓 ∈ 𝑅 𝑊 ×𝐻 ×𝐶 to 𝑋 𝑟 𝑓 ∈ 𝑅 𝑊 𝐻 ×𝐶 and 𝑋 𝑏2 ∈ 𝑅 𝑊 ×𝐻 ×𝐶 to 𝑋 𝑟 𝑏2 ∈ 𝑅 𝑊 𝐻 ×𝐶 . Then compute the similarity between feature maps:EQUATION)Using the obtained similarity image B, weight 𝑋 𝑟 𝑏2 , then reshape it to obtain the weighted feature map 𝑋 𝐶𝑆𝐴𝑀 :EQUATION)Controlling the generation of desired images in a controllable manner poses a challenging task for diffusion models. Especially when the objective is to obtain harmonious foreground images, it is crucial to ensure that the foreground and background share the same lighting distribution while preserving the content and structural information of the foreground objects. With the introduction of CLIP technology, text-guided diffusion models offer some controllable guidance. However, we recognize that images often provide more information than long texts. Therefore, in this module, we use compsite images with coarse shadows as conditions to guide the controllable generation of the diffusion model. Diffusion model generates an image 𝑥 0 by denoising a random image following a Gaussian distribution 𝑥 𝑇 ∼ N (0, I) progressively closer to the data distribution through multiple denoising steps , ObjectStitch [42] , DucoNet [44] and the SGDiffusion [26] , ARshadowGAN [25] and ground truth, respectively.𝑥 𝑇 -1 , ..., 𝑥 0 . Diffusion model is divided into forward diffusion and inverse denoising phases. Forward process. To construct training data, the forward process involves adding noise perturbations to the training image 𝑥 0 to generate noisy data 𝑥 1 , ..., 𝑥 𝑇 :EQUATIONwith 𝜖 ∼ N (0, I), and ᾱ𝑡 = 𝑡 𝑠=0 𝛼 𝑠 = 𝑡 𝑠=0 (1 -𝛽 𝑠 ). Reverse process. The reverse process aims to derive the posterior distribution for the less noisy image 𝑥 𝑡 -1 given the more noisy image 𝑥 𝑡 using the denoising network 𝑓 𝜃 :EQUATIONIn addition to adjusting the lighting effect of foreground objects, there are also issues with predicting rough shadows in the previous stage, which require further refinement through the network. We have observed the following issues:• The shape of the shadows generated in the previous stage is unrealistic. • The lighting of the foreground is inconsistent with the background image.We propose a method that uses composite images with coarse foreground object shadow (y) as conditional guidance to generate harmonized foreground object with realistic cast shadow. We train the denoising network 𝑓 𝜃 to predict 𝑥 0 instead of the noise 𝜖:EQUATIONFollowing [15] , our harmony diffusion model objective function isEQUATIONwhere 𝑥 𝑔𝑡 is the real image, m is the predicted foreground objectshadow mask. Considering that we need to iteratively optimize the shadow area, we also need to calculate the loss between the foreground object-shadow mask 𝑚 𝑓 and the generated foregroung object-shadow mask 𝑚 𝑡 for our method :EQUATIONTherefore, the total loss can be formulated as:EQUATION 
experiments 5.1 experimental setups: 
 The proposed method is implemented using PyTorch, and training is performed using two GeForce RTX 3090. The training epoch is set to 1000. We utilize the Adam optimizer with a momentum of (0.9, 0.999). The initial learning rate is set to 0.9. Following, we employ the Kaiming initialization technique to initialize the weights of the proposed model, and use a 0.9999 exponential moving average (𝐸𝑀𝐴) throughout all experiments. We adopt a U-Net architecture similar to the denoiser 𝜖 𝜃 in [10] . Training is carried out with 200 diffusion steps T and a noise schedule 𝛽 𝑡 that linearly increases from 0.0001 to 0.02, and inference is performed with 200 steps.We evaluated the performance of our method on IH-SG for object harmonization and shadow generation tasks. We resized the input and ground truth images to a size of 256 × 256 pixels. We calculated the Root Mean Square Error (RMSE), the Structural Similarity Index (SSIM) , fMSE, fSSIM for the generated images. And fMSE (resp., fSSIM) means MSE (resp., SSIM) within the foreground regions. In general, smaller values of RMSE and fMSE and larger values of SSIM and fSSIM indicate better quality of the generated images.We compare with following methods: DIH-GAN [1] , ObjectStitch [42] , DucoNet [44] , SGDiffusion [26] and ARshadowGAN [25] . Quantitative comparison. Table 1 reports the comparison results on IH-SG test set. It can be observed that our method achieves the best quantitative results across all four evaluation metrics. This is mainly because existing image harmonization methods struggle to generalize well to outdoor real-world datasets, while existing shadow generation methods either rely on simple estimations of foreground shadow masks or directly generate shadows using learned data distributions. Such inaccurate estimations often lead to inferior results. In contrast, our method leverages the coarse shadow prediction module (SP) to effectively utilize background information, and the harmonization diffusion model can better guide the lighting editing of inserted objects, refine features, bridge the lighting gap between inserted objects and background environments. Additionally, by iteratively refining shadow regions, our method achieves more realistic shadow effects closer to real images.Visual comparison. We provide some visual comparison results in Figure 6 . It can be observed that our method not only achieves lighting variations across different scenes but also achieves the best visual effects of realistic shadows. Among these competing methods, for ARShadowGAN, it is difficult to edit object lighting, and the generated shadows are not accurate in shape and direction. can harmonize foreground objects and generate realistic and reasonable cast shadows.We study the impact of image condition y, shadow prediction module (SP), and channel-spatial cross-attention (CSCA) mechanism of our method on test images from IH-SH. The results are shown in Table 2 , Figure 8 . We visualized the coarse shadow predict module in Figure 9 . It can be observed that model can effectively focus on the relevant areas in the background image, such as background objects and their shadows, and predict approximately correct shadows.To demonstrate the effectiveness of image conditions, we removed the guidance from images, denoted as "w/o y". The performance of "w/o y" is inferior compared to other models, indicating that utilizing image-condition guidance better preserves content structural information.To investigate the necessity of the coarse shadow prediction module SP, we removed this module, referred to as "w/o SP". It can be observed that without the SP module, there is a slight deficiency in shadow generation, and even the direction may be inaccurate. The performance of "w/o SP" is inferior to that of the full module, demonstrating the advantage of extracting background information and estimating coarse shadow regions.To demonstrate the effectiveness of the CSCA attention mechanism, the module was removed and replaced with a CAI layer [16] , known as "w/o CSCA". The results are not as good as the entire model, indicating that this mechanism can help the model generate more realistic images.Comparison on iHarmony4 [6] dataset. The results in Figure 7 demonstrate the applicability of our method in image harmonization tasks. It can be observed that DucoNet [44] and CDTNet [5] do not effectively transfer low-level illumination to the foreground, while our method achieves the best results. Our method can bridge the lighting gap between foreground objects and background environments while preserving the structural information of foreground objects, achieving lighting effects closer to ground truth (GT) images without changing the structure and detail of the foreground objects.Comparison on DESOBAv2 [26] and shadowAR [25] dataset. We perturbed the foreground objects in the DESOBAv2. As depicted in Figure 11 , the results demonstrate that our method can learn illumination information in the background to generate harmonious foreground objects and shadows for foreground objects. However, it is also observed that the generate shadows are somewhat unrealistic in few cases.Limitations: As depicted in Figure 11 , the proposed method has been successfully applied to image harmonization and shadow generation tasks in various environments. However, there are also some challenges. Firstly, our method faces challenges in generating non-planar cast shadows. This is because generating non-planar shadows requires more information, such as object geometry and environmental depth information, and the shadow generation results may be affected by mutual object occlusion. 
conclusion: 
 In this work, we have introduced a diffusion model-based method to edit the lighting of foreground objects and generate visually reasonable cast shadows with preserving the structure of the image. In addition, we have proposed a large-scale high-quality outdoor real-world dataset IH-SG for image harmonization and shadow generation tasks. Our future work is to solve the generation of non-planar cast shadows of foreground objects.  
abstract: 
 We propose a method for light and shadow editing of outdoor disharmonious composite images, including foreground harmonization and cast shadow generation. Most existing work can only perform foreground appearance editing tasks or only focus on shadow generation. In fact, lighting not only affects the brightness and color of objects, but also produces corresponding cast shadows. In recent years, diffusion models have demonstrated their strong generative capabilities, and due to their iterative denoising properties, they have a significant advantage in image restoration tasks. But it fails to preserve content structure of image. In this purpose, we propose an effective model to tackle the problem of foreground light-shadow editing. Specifically, we use a coarse shadow prediction module (SP) to generate coarse shadows for foreground objects. Then, we use the predicted results as prior knowledge to guide the generation of harmony diffusion model. In this process, the primary task is to learn lighting variation to harmonize foreground regions. The secondary task is to generate high-quality cast shadow containing more details. Considering that existing datasets do not support the dual tasks of image harmonization and shadow generation, we construct a real outdoor dataset, IH-SG, covering various lighting 
 
 Appendix:  


Decision: This paper receives a mixture of accepts and rejects. The negative conerns are mainly about insufficient experiments and flawed visual results. The authors provide the requested experimental results in the rebuttal, which verifies the effectiveness of the proposed method. Two reviewers mention that the generated shadows do not perfectly match the object shapes, and question the effectiveness in complex/occluded scenes. Since shadow generation is a very challenging task and there is still much room of improvement, the imperfect results cannot be solid reason for rejection. Considering the dataset contribution and effective method design, I tend to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a new method for light and shadow editing of outdoor composite images, aiming to achieve both foreground harmonization and realistic cast shadow generation. This is an important problem as lighting not only affects the brightness and color of objects but also produces corresponding shadows. Existing methods tend to focus either on foreground appearance editing or solely on shadow generation, while the paper addresses both aspects within one framework. 
The proposed approach is implemented through two steps:  a coarse shadow prediction module to generate initial shadows for foreground objects and a refined module to generate high-quality cast shadow containing more details.
strength_weakness: strength_weakness: The paper deals with foreground appearance editing and  shadow generation tasks with in one framework. The paper proposes an integrated method that tackles both foreground harmonization and cast shadow generation in outdoor disharmonious composite images.

The paper utilizes diffusion models, which have demonstrated strong generative capabilities, especially in image restoration tasks. By leveraging the iterative denoising properties of diffusion models, the method can generate harmonious foreground images and realistic cast shadows.

A coarse shadow prediction module (SP) is used to generate coarse shadows for foreground objects. These predicted results serve as prior knowledge to guide the generation of the harmony diffusion model, resulting in improved shadow generation.

A new dataset called IH-SG is delivered in the paper, which enriches performance comparisons in the field.
limitations: limitations: The paper deals with foreground appearance editing and  shadow generation tasks with in one framework. But the paper does not provide details on the specific model architecture used, including the network design, loss functions, and training procedures. Without these details, it is challenging to replicate the method or analyze its strengths and weaknesses.

Some hyperparameters such as in equation (16)  are chosen emperically without giving any interpretation.
paper_summary: paper_summary: This paper addresses the problem of foreground light-shadow editing. Specifically, a coarse shadow prediction module (SP) is introduced to generate coarse shadows for foreground objects. Then, the predicted results as prior knowledge to guide the generation of harmony diffusion model. Moreover,  the authors construct a real outdoor dataset, IH-SG, covering various lighting conditions. Experiments on IH-SG demonstrate its effectiveness.
strength_weakness: strength_weakness: This paper constructs a real outdoor dataset, IH-SG for foreground light-shadow editing.
limitations: limitations: 1.	The technical contribution of methodology is weak. The shadow prediction module is widely used in shadow generation, and the diffusion model has been adopted in image harmonization.
2.	The main experiment is only conducted on the constructed IH-SG. The authors only provide several visual results in section 5.5 on other datasets. However, the authors claim that extensive experiments conducted on both public datasets and our IH-SG dataset demonstrate the effectiveness of our method (lines 84-85, lines 181-182). It is overclaimed. The quantitative results on DESOBAv2 and DESOBAv2 are not provided.
paper_summary: paper_summary: This paper proposes a real outdoor dataset for image harmonization and shadow generation tasks. Propose a coarse shadow prediction network to predict cast shadows for foreground objects.Based on this dataset, it utilizes predicted prior with conditional diffusion model to generate higher quality shadows.
strength_weakness: strength_weakness: 1.	The structure of this paper is clear and logical.
2.	This paper presents a new approach to shadow generation tasks and provides a new and practical dataset based on their method.
3.	The shadows generated by the method proposed in this article are better than those produced by existing methods.
limitations: limitations: 1.	Based on the visual results presented in the paper, although the generated images are generally better than other methods, the overall visual quality of the results is not that good. Some high-frequency details are distorted and predicted shadows are somewhat unregular.
2.	While the network is clear, it appears rather simplistic. The application of the diffusion model seems somewhat conventional, and it's noticeable that the generated images lack texture details.
3.	The experiments are rather straightforward. Besides demonstrating the adaptability of their method, there is a lack of evidence showcasing the advantages of their proposed dataset. Additionally, there are few ablation experiments.
paper_summary: paper_summary: The paper introduces an innovative approach to harmonizing light and generating shadows in outdoor composite images. Unlike prior works that tackle either foreground editing or shadow generation in isolation, this research addresses the interplay between lighting and shadow casting comprehensively. The authors leverage the robust capabilities of diffusion models, particularly their iterative denoising properties, to restore images while maintaining the content structure. A key contribution is the construction of the IH-SG dataset, which provides a diverse set of lighting conditions. The extensive experimental evaluation demonstrates the method's superiority over existing techniques.
strength_weakness: strength_weakness: 1.The creation of a real-world outdoor dataset IH-SG that facilitates dual tasks of image harmonization and shadow generation. 2.A new image light-shadow editing method that achieves both controllable foreground harmonization and reasonable cast shadow generation. 3.Comprehensive experiments on public datasets and the newly created IH-SG dataset, which highlight the effectiveness of the proposed method.
limitations: limitations: 1.Lack of further elaboration on dataset creation: more details on the dataset creation process, such as the variety of scenes, objects, and lighting conditions included, would be beneficial. Additionally, insights into any potential biases in the dataset and how they might impact the model's performance could be valuable. 2.How's the generalizability to diverse environments? The paper demonstrates impressive results on IH-SG dataset. However, it would be beneficial to understand how the model performs on a wider range of environments, particularly those with varied and non-uniform lighting conditions not represented in the current dataset. Could the authors provide insights or additional experiments on the model's robustness in such scenarios?
paper_summary: paper_summary: This paper proposes a method for light and shadow editing of outdoor disharmonious composite images, including foreground harmonization and cast shadow generation. The proposed method can achieve controllable harmonization of foreground regions and a reasonable generation of cast shadows. Furthermore, this paper also proposes a new outdoor real-world dataset for image harmonization and shadow generation tasks. The experiment results show that the proposed method can generate shadows well and obtain good harmonization results.
strength_weakness: strength_weakness: 1. This paper proposes a new outdoor real-world dataset for image harmonization and shadow generation, which is valuable for future research.
2. The paper is well-written and easy to understand.
limitations: limitations: 1. The shapes of the shadows in the results presented in the paper do not closely match the shapes of the objects， such as Fig1.
2. The experiments and results presented in the paper are based on a resolution of 256x256. Can the model generalize to higher resolutions? Will the performance degrade at larger resolutions?
3. Introducing a new dataset should be one of the major contributions of this paper. However, there is no analysis of the dataset, such as data distribution and the overall size of the dataset. Why is this information missing?
4. I am curious about how the proposed method performs in complex cases, such as when the inserted objects overlap with background objects or when the background lacks clear shadow references.