DiffHarmony++: Enhancing Image Harmonization with Harmony-VAE and Inverse Harmonization Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:01:33

Abstract: Latent diffusion model has demonstrated impressive efficacy in image generation and editing tasks. Recently, it has also promoted the advancement of image harmonization. However, methods involving latent diffusion model all face a common challenge: the severe image distortion introduced by the VAE component, while image harmonization is a low-level image processing task that relies on pixel-level evaluation metrics. In this paper, we propose Harmony-VAE, leveraging the input of the harmonization task itself to enhance the quality of decoded images. The input involving composite image contains the precise pixel level information, which can complement the correct foreground appearance and color information contained in denoised latents. Meanwhile, the inherent generative nature of diffusion models makes it naturally adapt to inverse image harmonization, i.e. generating synthetic composite images based on real images and foreground masks. We train an inverse harmonization diffusion model to perform data augmentation on two subsets of iHarmony4 and construct a new human harmonization dataset with prominent foreground objects. Extensive experiments demonstrate the effectiveness of our proposed Harmony-VAE and inverse harmonization model.  The code, pretrained models and the new dataset will be made publicly available.

Main paper: 
 introduction: 
 Image composition is a crucial technique in digital editing, where a new image is created by combining the foreground of one image with the background of another. This process, widely applied in fields such as advertising and entertainment, often encounters a challenge: the merged image may appear unrealistic due to mismatches in lighting and color between the foreground and background. To solve this problem, image harmonization is employed. It involves modifying the foreground to ensure visual consistency with the background, focusing on aligning color, illumination, and texture, without changing the original content or meaning. Recent advancements in deep learning [45, 11, 17, 41] have significantly facilitated image harmonization.The latent diffusion model (LDM) [37] is gaining increasing attention and has established new SoTA in the field of image generation. It can swiftly transfer to downstream image-to-image translation tasks such as image editing [1, 23] and restoration [33, 47] . As image harmonization can be categorized as an image-to-image translation task, the latent diffusion model is also suitable for doing image harmonization. For example, Li et al. [27] follow the network architecture in ControlNet [51] and adapt pretrained latent diffusion model, i.e. Stable Diffusion, to perform image harmonization, but they ignore the image distortion problem inherent in VAE decoding process, resulting in unsatisfactory model performance. Zhou et al. [52] adopt a two-stage approach. The first stage finetunes the denoising UNet, while the second stage involves designing a refinement module to alleviate the image distortion issue. It finally achieves SoTA performance. However, this approach exhibits several limitations: 1) The second-stage training relies on the results of the first stage, necessitating retraining of the second stage upon updates in the first stage. 2) Introducing an independent UNet model complicates the harmonization process. 3) It's limited to usage at fixed resolutions.To solve aforementioned limitations, we propose Harmony-VAE, which incorporates composite images as additional information into the decoding process of the VAE, enabling the VAE decoder to reconstruct images with realistic details. Our motivation stems from the highly precise object shape information present in composite images, which can complement the correct foreground appearance and color information contained in denoised latents. The training of Harmony-VAE is independent of fine-tuning denoising UNet; it reconstructs real images with composite images as condition, significantly reducing training costs, and it can improve the performance of all LDM-based harmonization models. Furthermore, although trained only on 256px images, Harmony-VAE significantly enhances the decoding quality of higher resolution images, demonstrating strong generalization capabilities. We call DiffHarmony equipped with Harmony-VAE as DiffHarmony++.The latent diffusion model enhanced by our proposed Harmony-VAE, with simple modifications, can be adapted to construct a model that performs inverse image harmonization, serving the purpose of data augmentation. Inverse image harmonization refers to the generation of synthetic composite images from real images and foreground masks. The outcomes should not be unique, as the content of the images may be influenced by various factors such as weather and lighting conditions. This process can be modeled as a one-to-many image-to-image translation task, and employing a diffusion model to handle it is a very natural choice. Based on DiffHarmony++, we train an inverse harmonization diffusion model on iHarmony4 [10] . This model enables virtually unlimited expansion of harmonization data. Leveraging the inverse harmonization model, we perform data augmentation on two smaller subsets of iHarmony4, namely Hday2night and HFlickr. Compared to training solely on the original data, training with augmented data greatly enhances performance on these two datasets.Another significant advantage of our proposed inverse harmonization model is to automatically construct new image harmonization datasets, which has long been a challenge in the field of image harmonization. The creation of datasets like RealHM [20] often needs extensive manual effort and expertise in digital image processing, proving to be time-consuming and labor-intensive. iHarmony4 employs automatic color transfer algorithms to generate synthetic composite images, but still need manual screening to filter out unreasonable data. Leveraging the inverse harmonization model, we generate a substantial number of synthetic composite images based on the imaterialist [12] dataset and train a harmony classifier to identify most unharmonized images, culminating in the creation of the Human Harmony dataset. Training harmonization models on this newly constructed dataset, both qualitative and quantitative evaluations affirm that the proposed inverse harmonization diffusion model is a highly promising approach for building new datasets.Our contributions can be summarized as follows:• We propose Harmony-VAE to tackle the image distortion problem inherent in VAE decoding process, which is the most challenging aspect of applying LDM to image harmonization tasks.  
related works 2.1 image harmonization: 
 Image harmonization is a critical task in image composition [34] , aimed at achieving visual consistency between the foreground and background of composite images. Early efforts were primarily centered around traditional color matching algorithms [49, 43, 44, 50, 7] . These methods focused on aligning the low-level color statistics between the foreground and background, utilizing techniques such as shifting, scaling, and histogram matching. For example, Lalonde et al. [26] recolor image regions for realistic compositing by studying the color statistics of a large dataset of natural images and looking at differences in color distribution in realistic and unrealistic images. With the advent of deep learning, a surge of supervised deep harmonization methods came out [45, 11, 17, 41, 36, 46, 48, 53, 20, 2, 3, 4] . Tsai et al. [45] propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. Cun et al. [11] learn the feature map in the masked region and the others individually with a novel attention module named Spatial-Separated Attention Module. The application of domain translation or style transfer techniques to image harmonization [8, 16, 30, 10] offers a creative means of reconciling discrepancies between the foreground and background by conceptualizing different illumination conditions as distinct domains or styles. Cong et al. [10] translate foreground to the same domain as background with a domain verification discriminator. Cong et al. [8] use a domain code extractor to capture the background domain information to guide the foreground harmonization, which is regulated by well-tailored triplet losses. In the mean time, the integration of Retinex theory [15, 14, 13] into image harmonization has opened new pathways by decomposing images into reflectance and illumination maps, and also the development of deep networks predicting color transformation [9, 22, 29, 49 ] signifies a balance between efficiency and effectiveness, streamlining the harmonization process without sacrificing quality.Jiang et al. [20] construct RealHM dataset with 216 image pairs by manually adjusting the foreground according to the background, which is time-consuming, labor-intensive, and unreliable. Another line of work is collecting a set of foreground images captured in different illumination conditions, followed by replacing one foreground with another counterpart, for example Transient Attributes Database [25] (101 sets, in which each set has well-aligned images for the same scene captured in different conditions). Cao et al. [3] and Guo et al. [15] try to construct harmonization dataset by varying the lighting condition of the same scene using 3D rendering techniques, however the rendered images have large domain gap with real images so the trained model is hard to directly applied to real test images. Some other works [45, 11, 10] adopted an inverse approach, i.e., adjusting the foreground of real image to create synthetic composite image. The representative, iHarmony4 dataset utilize automatic color transfer for foreground adjustment followed by aesthetic predictor and binary CNN classifier for filtering (also at last need manual screening). Niu et al. [35] learns a VAE conditioned on ground-truth real image and foreground mask to predict color transformation LUT to adjust foreground part in real image, producing new synthetic composite images automatically.Diffusion models are adept at generating realistic images from random noise, showcasing unparalleled performance in image synthesis. Ho et al. [19] introduced Denoising Diffusion Probabilistic Models (DDPMs) employing a Markovian diffusion process. This method progressively adds noise to an image until it becomes pure noise. Then, a deep neural network trained to predict this noise reverses the process, creating a new image from the noise. Alternatively, a non-Markovian diffusion process [42] is proposed to offer a quicker and more adaptable solution than the Markovian method in DDPMs. Due to the remarkable ability to create lifelike images, diffusion models have been extensively used in various image synthesis applications by researchers. For example, Palette [40] , a conditional diffusion model set new standards in image-to-image translation tasks including colorization, inpainting, uncropping, and JPEG restoration. RePaint [32] utilized a pre-trained unconditional DDPM as a basis for generation, conditioning the process by sampling from unmasked regions of the provided image data. Sahak et al. [39] introduced SR3+, a diffusion model that set new benchmarks in the blind super-resolution task. Diffusion models in latent space have recently garnered unprecedented attention. Latent Diffusion Model (LDM) [37] is proposed to serve as the theoretical foundation for Stable Diffusion. Numerous works based on Stable Diffusion targeted on solving imageto-image translation tasks, and some have applied latent diffusion model framework to the image harmonization task (which can be categorized as image-to-image translation). For instance, Appearance Consistency Discriminator [27] is proposed to guide LDM and convert the generated image from RGB to HSV/HSL space to adjust the lightness channel. Chen et al. [6] achieved Zero-Shot Image Harmonization by incorporating Attention-Constraint Text and Content Retention modules into Stable Diffusion. Lu et al. [31] focused on using pretrained LDM to blend photographic objects into paintings, achieving artistically coherent composite images. Zhou et al. [52] proposed DiffHarmony, which adapted the Stable Diffusion inpainting variation for image harmonization and propose a refinement stage to address image distortion issues. 
harmony-vae 3.1 preliminary: 
 The Stable Diffusion (SD) model stands as a quintessential representation of latent diffusion model and serves as the cornerstone for our approach. It undergoes pre-training in two distinct stages employing a VAE (Variational AutoEncoder) and a denoising U-Net. In the first stage, it trains the VAE, where the encoder E first encodes the image I into a latent space, resulting the latent variable 𝑧 0 = E (I); subsequently the decoder D endeavors to reconstruct it into the original image, yielding the reconstructed image Î. In the second stage, the VAE is frozen and the goal is training a denoising U-Net 𝜖 𝜃 , which involves adding noise over 𝑡 steps to the latent variable 𝑧 0 to get noisy latent 𝑧 𝑡 (𝑡 ∈ [1,𝑇 ]), and updating the denoising U-Net with latent denoising loss, as formulated below:L = E 𝑧 0 ,𝑐,𝜖∼N (0,1),𝑡 ∥𝜖 -𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐)∥ 2 2 (1)Here, 𝜖 denotes the noise added to the latent variable 𝑧 0 at each noise step, 𝜖 𝜃 represents the denoising U-Net, receiving timestep 𝑡 and additional conditions 𝑐 (e.g., text, conditional images, masks, etc.) as input.During the inference process, pure noise 𝑧 𝑇 is sampled from a normal distribution. The model will iteratively employ 𝜖 𝜃 to estimate the noise for each denoising step 𝑡, progressively refining the latent variable 𝑧 𝑇 to ultimately attain denoised latent variable 𝑧 ′ 0 . Finally, the denoised latent variable 𝑧 ′ 0 is fed into the decoder D to generate the image. For intricate details on the training and inference of Stable Diffusion, please refer to [37] .In image harmonization task, we delineate the conceivable variables as follows: the input to image harmonization task comprises the composite image 𝐼 and the foreground mask image 𝑀 (where the foreground is represented as 1 in white, and the background as 0 in black), while the output is the harmonized image Ĩ . Ground-truth real image utilized in model training and evaluation is denoted as Î . The composite image 𝐼 can be divided into the foreground component 𝐼 𝑓 and the background component 𝐼 𝑏 . The objective of image harmonization task is to adjust the composite foreground 𝐼 𝑓 to produce the harmonized image Ĩ , which should closely resemble Î .When employing the latent diffusion model for image harmonization, a viable approach could be: during training, setting 𝑧 0 = E ( Î ), 𝑐 = (E (𝐼 ), down(𝑀)) (where down denotes the downsampling operation), in detailed words utilizing the encoded composite image and downsampled mask image as conditions (typically concatenated along the channel dimension) for denoising the noisy latent 𝑧 𝑡 . During inference, initialize from random noise 𝑧 𝑇 , progressively denoise it to obtain 𝑧 ′ 0 and decode it through D to yield Ĩ . This approach has already been proven effective in DiffHarmony [52] , and in our subsequent descriptions, we assume that the latent diffusion model part adopts the same architecture as it.When employing latent diffusion models for image harmonization task, a common obstacle to model performance is the image distortion problem caused by VAE [24] decoding process. Image harmonization falls within the domain of low-level image processing, demanding pixel-level accuracy of output images. However, VAE in Stable Diffusion, which decodes a single latent variable to an image, often results in distortion in subtle high-frequency textures or lead to the fabrication of content not present in the original image (as VAE fundamentally operates as a generative model).To address the aforementioned issue, we propose the Harmony-VAE, aiming at enhancing the quality of the output harmonized image Ĩ during the decoding stage of VAE by utilizing the input of the harmonization task, namely the composite image 𝐼 and foreground mask image 𝑀.Specifically, we incorporate an additional encoder, the conditional encoder E 𝑐 , to encode 𝐼 and 𝑀. During the encoding process, we obtain the final latent variables and intermediate features of the encoder. These features are fused into VAE decoder through skip connections (similar to UNet [38] ) and element-wise addition. The training objective of the Harmony-VAE is to reconstruct the real image Î with 𝐼 and 𝑀 as condition. This process can be mathematically formalized as: L = Î -D E ( Î ), E 𝑐 (𝐼, 𝑀) 
inverse image harmonization 4.1 inverse harmonization model: 
 The acquisition of training data for image harmonization task has always been a costly endeavor. Traditional procedures for constructing harmonization datasets often involve three main steps: selecting challenging compositing pairs, creating precise masks for foreground objects with distinct boundaries, and adjusting foreground appearance to match the background using tools like Photoshop [20] . These steps require a considerable amount of manual labor and expertise in image processing, making them challenging and time-consuming. The construction of the iHarmony4 [10] dataset employs a highly automated process, such as color transfer modules and classifiers to identify unreasonable data, but still requires manual screening.Diffusion models are initially applied to image generation and conditional image generation. Considering harmonization data generation (as in iHarmony4), given the foreground mask image 𝑀 and the real image Î , the output is the composite image 𝐼 . The resulting composite image 𝐼 is not unique because many variations can make the foreground appear incongruous with the background. This process can be modeled as a typical one-to-many conditional image-to-image translation task, hence employing diffusion models to generate harmonization data is a natural choice.Our approach is based on DiffHarmony++ to train an inverse harmonization model. The model takes the foreground mask image 𝑀 and the real image Î as input and outputs the composite image 𝐼 . We train it using the existing harmonization dataset iHarmony4.We can use the inverse harmonization model to generate additional training data for existing harmonization datasets. Specifically, we generate 𝐾 candidate composite images for each ( Î, 𝑀) data pair in the dataset and specify the blending ratio as 𝛾. Assuming the number of ( Î, 𝑀) data pairs in the original dataset is 𝑁 1 , and the total number of original composite images is 𝑁 2 , we randomly select 𝛾 𝑁 2 𝑁 1 newly generated composite images for each ( Î, 𝑀) data pair and add them to the training data. After blending, we train new harmonization models separately using the original dataset and the augmented dataset, comparing the results of the two experiments on oiriginal test set to observe the benefits of data augmentation (Experimental results show that our data augmentation strategy significantly improves model performance compared to training solely on the original data. Details can be found in the 5.3 section).The field of image harmonization has long lacked datasets specifically tailored to human portraits domains. These datasets have the potential to optimize product showcases and enhance user experiences, offering significant value and innovation opportunities for e-commerce, retail, and fashion design. However, for the reasons mentioned earlier, constructing such datasets has been challenging. Now, after verifying the effectiveness of our inverse harmonization model, we have the opportunity to construct a Human Harmony dataset in a cheap and fast way.Furthermore, the analysis in DiffHarmony [52] suggests that latent diffusion models may perform better on samples with large foregrounds. Portrait photography data typically features large We select imaterialist-fashion-2020-fgvc7 [12] as our initialization. It comprises a vast number of high-resolution portrait photographs, and the majority features large foregrounds objects (human body parts). Each image is paired with highly-detailed segmentation map, which can be used to construct accurate foreground mask.To ensure the quality of the generated harmonization data, we train a harmony classifier using the iHarmony4 dataset. Given an input image, the classifier outputs the probability of it belonging to composite images.Following Section 4.1, we generate 𝐾 candidate composite images for each ( Î, 𝑀) data pair. To ensure the quality of the generated images, we separately train a Harmony-VAE suitable for inverse harmonization model using the iHarmony4 dataset. Subsequently, we use the harmony classifier to classify the candidate set, selecting the image with the highest classification probability as the final composite sample. 
experiment 5.1 datasets and metrics: 
 5.1.1 iHarmony4 Dataset. We conduct experiments on the benchmark iHarmony4 [10] dataset, which consists of four sub-datasets: HCOCO, HFlickr, HAdobe5K, and Hday2night, with a total of 65,742 (7,404 for testing) pairs of composite and real images in the training (testing) sets. Following previous work [10, 41] , we merge the training set of the four sub-datasets into one entire training set and evaluate each sub-dataset individually.For the Human Harmony dataset, we filter the imaterial-fashion-2020-fgvc7 [12] dataset to remove images containing only products. After cleaning the dataset size is 29,106. We set all valid segmentation parts (except background) to 1 to construct foreground mask images. When building the Human Harmony dataset, we set 𝐾 = 10 and use the harmony classifier to select the image with the highest classification probability. We divide the Human Harmony dataset according to the same training/testing set ratio as the iHarmony4 dataset, resulting in a training set of size 26,157 and a testing set of size 2,946. We analyze the distribution of foreground ratios and plot cumulative distribution curves. We also simultaneously plot the curve of iHarmony4. From Figure 2 , it can be observed that approximately 70% of the images in the Human Harmony dataset have foreground ratios above 0.2, whereas in iHarmony4 this is less than 15%.In line with previous works [16] , our DiffHarmony++ and the baselines are evaluated using Peak Signalto-Noise Ratio (PSNR), Mean Squared Error (MSE), and foreground MSE (fMSE) calculated across RGB channels. fMSE is a specific evaluation metric that solely measures the MSE within the foreground region, gauging the success of foreground harmonization.During the training of Harmony-VAE, we use all iHarmony4 training data. We set lr = 1𝑒 -4, with warmup = 0.02, and then keep it constant. The training lasts 10 epochs. We use the AdamW optimizer, with weight_decay=0, 𝛽 1 = 0.9, and 𝛽 2 = 0.999. We save model weights using exponential moving average (EMA), with max_decay = 0.999. Images are resized to 256px during training. As for the diffusion model, we adopt the pre-trained DiffHarmony [52] directly. Following [52] , we use Euler ancestral discrete scheduler [21] to generate the samples in only 5 steps during inference.The implementation details of inverse harmonization model are mostly consistent with DiffHarmony++, except that we use ( Î, 𝑀) For the harmony classifier, we utilize pre-trained ResNet50 [18] model, taking the final 2048-dimensional features after pooling for linear probing. We train a binary classification head using all iHarmony4 training data.5.3.1 Results on iHarmony4. On the iHarmony4 dataset, we compare our approach with the following image harmonization methods: DoveNet [10] , BargainNet [8] , RainNet [30] , iS 2 AM [41] , D-HT [14] , SCS-Co [16] , HDNet [5] , Li 𝑒𝑡 𝑎𝑙 . [28] , GIFT [35] , DiffHarmony [52] , etc. In Table 1 , we report the average results for the four sub-test sets and the entire test set, which are either replicated from the original papers or reproduced using publicly available models. For the overall results on the entire test set, our method significantly outperforms previous SOTA methods. Our method achieves the best results on almost all sub-sets , except for the MSE on the HAdobe5k and Hday2night subsets.Our proposed model outperforms DiffHarmony because we utilize Harmony-VAE instead of the VAE from Stable Diffusion. In the process of image decoding, using only the regular VAE from Stable Diffusion can result in severe distortion. In contrast, our proposed Harmony-VAE can preserve more details. Our qualitative results are provided in Figure 3 Our experimental results are shown in Table 2 . "base" represents the results obtained by training only on the original dataset, while "augmented" represents the results obtained by training on the augmented dataset. For both datasets, we set the blending ratio 𝛾 = 1. All experiments use the same hyperparameter setting and train with the same total amount of data. We train enough steps to ensure convergence and then evaluate on the original test set. We perform inference only on 512px images and then scale the results to 256px for evaluation, as achieving optimal inference performance is not necessary for observing the effectiveness of data augmentation. The results show that data augmentation with inverse harmonization model significantly improves the model performance on both datasets, thus validating its ability to generate high-quality composite images.In Table 3 , we present a detailed comparison of the performance of DiffHarmony++ and HDNet on both the iHarmony4 dataset and the Human Harmony dataset. To ensure fairness, we stipulate that models directly utilize composite images as input during both training and inference. (Note: HDNet in its original implementation crops the background portion of real images as input background content, significantly improving its performance and resulting in unfair comparisons with other approaches.) Following HDNet [5] , we divide data into three ranges based on the ratio of the foreground region area and the entire image: 0% ∼ 5%, 5% ∼ 15%, and 15% ∼ 100%. We calculate metrics for each range respectively. We train HDNet with 512px images, denoting it as HDNet 512 . During testing, we utilize 1024px images as input and subsequently resize harmonized images to 256px for evaluation, maintaining consistent experimental settings with our approach.Our results on the iHarmony4 dataset indicate that when test samples have small foreground proportions (0% ∼ 5%), HDNet 512 outperforms DiffHarmony++. However, as the foreground proportion increases, DiffHarmony++ demonstrates increasingly superior performance. On the Human Harmony dataset, particularly on larger foreground proportions (15% ∼ 100%), DiffHarmony++ consistently outperforms HDNet 512 . Regarding its performance on samples with smaller foreground proportions, we speculate that the limited number of data samples leads to a significant variance in statistical metrics. Due to constraints in time and computational resources, further investigation into the underlying reasons for this phenomenon is deferred for future research.Qualitative results are shown in Figure 4 . Our approach often generates visually appealing outcomes that closely resemble the authentic real images.We conduct ablation study to validate the effectiveness of the components in our proposed Harmony-VAE. 4 : Ablation study of using Harmony-VAE at multiple inference resolutions. For easier comparison we add additional results of DiffHarmony plus refinement module at bottom and mark the corresponding lines in gray color.We do inference with and without Harmony-VAE at multiple resolutions. As shown in Table 4 , adding the Harmony-VAE results in an improvement in the overall performance. The benefit of introducing Harmony-VAE is more prominent when DiffHarmony uses lower image resolutions, as the Harmony-VAE and using higher resolution input both aim to address the issue of image distortion, and they complement each other. We also add additional results of DiffHarmony plus refinement module [52] at bottom and mark the corresponding lines in gray color. Comparing to using only Harmony-VAE, using only the refinement module and the cascaded use of Harmony-VAE and refinement module only bring negligible improvement. This suggests that employing Harmony-VAE alone to enhance DiffHarmony could achieves optimal performance, and 5 : Ablation study of our proposed zero init strategy and finetuning different parameters.From Table 5 , it can be observed that not using zero-initialized convolution severely damages the normal training of the model. The random initialization of convolution layers introduces nonnegligible perturbations to the original feature distribution when fused with the VAE decoder. From the fine-tuning parameter ablation experiments, it can be seen that fine-tuning only the conditional encoder already provides a considerable improvement in reconstruction performance, while unfreezing the decoder can further increase the benefits. 
conclusion: 
 In this paper, we have proposed the Harmony-VAE, aimed at leveraging the conditional information in image harmonization tasks to enhance the quality of image decoding of the VAE component in the latent diffusion model. Our proposed Harmony-VAE preserves finer details, effectively restoring severely damaged facial features, architectural patterns, small text during the decoding process. Furthermore, we have trained an inverse harmonization model which can synthesize new composite images based on real images and foreground masks. The substantial improvements observed on the Hday2night and HFlickr datasets attest to the efficacy of our model. Building upon this, we have subsequently constructed the Human Harmony Dataset, comprising samples with prominent foreground areas. Experimental results demonstrate the effectiveness of our inverse harmonization model of superiority of the LDM-based harmonization approach on samples featuring prominent foreground objects.  
 
 Appendix:  


Decision: This paper receives a mixture of reviews. It is important to tackle image distortion when using diffusion model for image harmonization, which is addressed in this paper. This paper also makes the first attempt at using diffusion model for data augmentation in the field of image harmonization. Although similar ideas have been explored in other fields like image inpainting, the contribution to the field of image harmonization is non-negligible. The rebuttal has addressed the concern of train/inference biases. The visual results in the supplementary can justify the higher quality of the results generated by the proposed method. The authors are suggested to cite and discuss the two related papers mentioned by Reviewer fxTh.

Reviews: review: paper_summary: paper_summary: The paper proposes Harmony-VAE, leveraging the input of the harmonization task itself to enhance the quality of decoded images. The paper employs diffusion models to inverse image harmonization, i.e. generating synthetic composite images based on real images and foreground masks. 
Extensive experimental results are given to demonstrate the effectiveness and performance of our proposed Harmony-VAE and inverse harmonization model.
strength_weakness: strength_weakness: The Harmony-VAE model is proposed to leverage the image harmonization task itself to enhance the quality of decoded images. This approach utilizes the precise pixel-level information contained in the composite image to complement the denoised latents, potentially improving the foreground appearance and color information.

An inverse harmonization diffusion model is propsed to to automatically construct new image harmonization datasets, which has long been a challenge in the field of image harmonization.

Experimental results are given to demonstrate the effectiveness of their proposed Harmony-VAE and inverse harmonization model, validating the proposed approach can achieve state-of-the-art performance on various image harmonization tasks.

A new dataset called Human Harmony dataset is delivered in the paper.
limitations: limitations: The paper lacks details about the exact network architecture, hyperparameters, and training procedures used for the Harmony-VAE and inverse harmonization models.  The hyperparameters in the implementation details are given emperically without any validation. 

The evaluation methods used in the paper are not so convincing. We cannot see apparent differences of qualitative results in fig 3 and 4. The evaluation metrics used in paper are Peak Signalto-Noise Ratio (PSNR), Mean Squared Error (MSE), and foreground MSE (fMSE),  how does these metrics are related to image harmonization?
paper_summary: paper_summary: The paper introduces a sophisticated approach to improve the visual coherence of composite images. The authors propose Harmony-VAE to refine the decoding process of a latent diffusion model, addressing the distortion issues inherent in VAEs. Additionally, they present an inverse harmonization model for data augmentation, which creates synthetic composite images from real images and foreground masks. The paper also introduces a new Human Harmony dataset, enriching the multimedia community with resources for image harmonization tasks.
strength_weakness: strength_weakness: (1).Theoretical grounding: the paper is theoretically sound, with a clear explanation of how Harmony-VAE leverages additional input to improve image decoding. (2).Empirical validation: The empirical methodology is robust, with extensive experiments conducted on the iHarmony4 dataset and the newly created Human Harmony dataset, demonstrating the model's effectiveness. (3).Significance and novelty: the contributions of this paper are significant and novel. The Harmony-VAE presents a new way to tackle the distortion problem in VAEs, which is a critical issue in image harmonization tasks. The introduction of an inverse harmonization model for data augmentation is innovative.
limitations: limitations: (1).Depth of structural design analysis: The paper could benefit from a more in-depth analysis of the structural design choices. While the Harmony-VAE and inverse harmonization model are novel, there is a lack of detailed justification for the specific architectural decisions made. (2).Exploration of hyperparameter sensitivity: The paper does not delve into the sensitivity analysis of the hyperparameters used in the models. Understanding how changes in hyperparameters affect the outcomes is crucial for the robustness of the proposed methods.
paper_summary: paper_summary: This paper designs and trains a Variational Autoencoder (VAE) with an encoding-reconstruction task and then utilizes it in the Diffusion Model's VAE decoder to improve DiffHarmony's ability, namely DiffHarmony++. Moreover, this work is based on DiffHarmony++ to train an inverse harmonization model, which can produce composite images from real images and foreground masks.
strength_weakness: strength_weakness: This work automates the construction of composite images using an inverse harmonization model and validates its effectiveness on small datasets.
limitations: limitations: 1. The proposed method's innovation is limited, as it primarily involves technical improvements to DiffHarmony, with its performance mainly derived from the effectiveness of the diffusion model. Compared to DiffHarmony, the performance improvement of DiffHarmony++ is not significant. Additionally, there is a discrepancy between the performance of DiffHarmony and the original paper, which is not explained.
2. The approach of automatically generating composite images is not novel, as the idea of using generative models to augment training data has been repeatedly discussed by research groups since the advent of GANs. Moreover, this paper only validates the effectiveness of generated data on a small dataset.

This paper lacks novelty and significant contributions, which are crucial for acceptance in ACM MM  papers.
paper_summary: paper_summary: This paper proposed an incremental VAE training strategy to improve the latent diffusion-based harmonization. The proposed Harmony-VAE could improve the high-frequency details for the original decoded results. This model can be also trained with inverse harmonization as a data augmentation for harmonization. Experiments show the effect of this VAE to preserve some foreground details.
strength_weakness: strength_weakness: 1. The proposed method is straightforward and easy to follow.
2. The idea is simple but effective for latent diffusion-based manners.
limitations: limitations: 1. The novelty is constrained. This work is more like an incremental work based on DiffHarmony to eliminate the bias of VAE decoder.
Moreover, the limitation of VAE decoder has been widely discussed, but this paper miss some important related works, such as [1][2].

[1] Zhu Z, Feng X, Chen D, et al. Designing a better asymmetric vqgan for stablediffusion[J]. arXiv preprint arXiv:2306.04632, 2023.

[2] Wang Y, Cao C, Fu Y. Towards Stable and Faithful Inpainting[J]. arXiv preprint arXiv:2312.04831, 2023.

2. The model designed in Fig1 suffers from train/inference biases. Because the training of Harmony-VAE uses GT inputs, while the inference uses denoised ones instead. The authors did not discuss how to address this issue.

3. The illustration of this paper is not well formulated. For example, it is really hard to identify which one is better in Fig.3 without the foreground mask. Moreover, there are no clear teasers to present and claim the contribution of this paper.