Interactive Portrait Harmonization
Authors: Jeya Maria Jose Valanarasu
HE Zhang
Jianming Zhang
Yilin Wang
Zhe Lin
Jose Echevarria
Yinglan Ma
Zijun Wei
Kalyan Sunkavalli
Vishal Patel
Venue: ICLR 2023 poster (2023)
Keywords: harmonization
image editing
low-level vision

Abstract: Current image harmonization methods consider the entire background as the guidance for harmonization. However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization. To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected region in the reference image instead of the entire background. A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed. Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region. This framework provides more control to the image harmonization pipeline achieving visually pleasing portrait edits. Furthermore, we also introduce a new dataset carefully curated for validating portrait harmonization. Extensive experiments on both synthetic and real-world datasets show that the proposed approach is efficient and robust compared to previous harmonization baselines, especially for portraits.

Published as a conference paper at ICLR 2023
INTERACTIVE PORTRAIT HARMONIZATION
Jeya Maria Jose Valanarasu' 1 "He Zhang" Jianming Zhang Yilin Wang". Zhe Lin Jose Echevarria', Yinglan Ma², Zijun Wei", Kalyan Sunkavalli", Vishal M. Patel I Johns Hopkins University, 2 Adobe Research
ABSTRACT
Current image harmonization methods consider the entire background as the guid- ance for harmonization. However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization. To en- able flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected region in the reference image instead of the entire background. A new flexible framework that allows users to pick certain regions of the background im- age and use it to guide the harmonization is proposed. Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss optimally match the color/luminance conditions between the composite fore- ground and select reference region. This framework provides more control to the image harmonization pipeline achieving visually pleasing portrait edits. Further- more, we also introduce a new dataset carefully curated for validating portrait harmonization. Extensive experiments on both synthetic and real-world datasets show that the proposed approach is efficient and robust compared to previous harmonization baselines, especially for portraits. The code can be found here: Parrrieiitooootttttaaaaaaniiiioo
INTRODUCTION
With the increasing demand of virtual social gathering and conferencing in our lives, image har- monization techniques become essential components make the virtual experience more engaging and pleasing. For example, if you cannot join wedding or birthday party physically but still want to in the photo, the first option would be to edit yourself into the image. Directly compositing yourself into the photo would not look realistic without matching the color/luminance conditions. One possible solution to make the composition image more realistic is to leverage existing image harmonization methods Cong (2020 2021): Cun & Pun (2020): Ling al. (2021) Jiang al.
2001
Guo et al. 2021b): sai et al. (2017).
Most previous works focus more general image harmonization setup, where the goal is match a foreground object to a new background scene without too much focus on highly retouched portrait. However, when we conduct surveys among professional composition Photoshop/Affimn realized that portrait harmonization is the most common task of image editing in real-world scenario and professional settings. This makes portrait harmonization the most important use case of image harmonization. We note that previous harmonization works have not focused on addressing portrait harmonization on hal-world data. In this work, we aim to explore a better solution obtaín realistic and visually pleasing portrait harmonization for real-world nigh-resolution edited images.
Jusers,
One common question that pops up when we demo existing image harmonization workflow to these professional users is: 'How could we choose certain person as reference when we do harmo- nization with existing workflow ?". The workflow design existing state-of-the-art harmonization methods Cong al. (2020) 2021): Ling et al. (2021); Guo et al. (2021b) limits the capability for user to choose any person/region as reference during the harmonization process. These frameworks designed such that they just take in the composite image and foreground mask as input thus offering no specific way to help the user to guide the harmonization. Certain frameworks such as
Work done while doing an internship at Adobe https://affinity::
c.w.m/n gb/phtto/

---

Published as a conference paper at ICLR 2023
Refevence
Compottte
Foreground
Region
Composite
Ponegroand
Composite Image
Mask
Background Hamonization SOTA
Interactive Harnonization
(Ours)
Figure 1: Testing with in-the-wild portrait composites. Top row- Professional Studio Portrait, Bot- tom row- Non-studio portrait. It can be observed that the current SOTA harmonization method Ling al. (202I) fails give realistic results it tries to match the appearance of foreground to the entire background. Our proposed interactive barmonization framework produces a visually pleasing portrait harmonization as it matches the appearance of the original portrait in the reference region instead of the entire background as selected by the user.
Bargain-Ner Cong al. (2021) have the flexibility be tweaked and converted interactive harmonization. However, from our experiments we find that they are not robust enough to perform realistic portrait harmonization.
Furthermore, professional portraits are mostly shot at studios where screens constitute the back- ground. These screens offer little no information in harmonizing new composite when edited into the photo. This causes current harmonization methods to produce unstable results (see first row of Figure D as they have not been trained to perform harmonization with screens as background. Also, portraits which are captured by everyday users usually contain background of spatially vary- ing luminance characteristics. Using the entire background to guide harmonization here can result in undesirable outcomes (see second row of Figure I as harmonization depends on the location where the foreground is composited to.
To this end, we introduce Interactive harmonization- more realistic and flexible setup where the user can guide the harmonization. The user can choose specific regions in the background as a refer- ence to harmonize the composite foreground. From professional stand-point, this is much-needed feature as could enable a lot of control for editors. this work, we propose a new interactive por- trait harmonization framework that has the flexibility take in reference region provided by the user. We use an encoder-decoder based harmonization network which takes in the foreground com- posite region as input and a style encoder which takes the reference region as its input. The reference region to guide the harmonization is selected by the user and can be a person, and object even just some part of the background. However, it can be noted that in portrait editing it is very common to choose another person in the picture reference to obtain effective harmonization. We also use style encoder that extracts the style code of the reference region and injects it the harmonized foreground. We carefully align the style information with foreground while also pre- serving the spatial content using adaptive instance normalization [uang Belongie (2017) layers at decoder. To make the harmonization look more realistic it is important to optimally match the lu- minance, color, and other appearance information between the reference and foreground. To match these characteristics in manual photo editing, professional photography tics of the highest (highlight), lowest (shadow) and average (mid-tone) of luminance points between the composite foreground image and the reference region. Hence, propose new luminance matching loss that is inspired by professional users [] In the proposed loss, we match the highlight, mid-tone and shadow points between the reference region and foreground composite region.
usually match statis-
Current publicly available datasets are developed only for background harmonization problem. So, curate new IntHarmony dataset to enable training for interactive harmonization utilizing aug-
2

---

Published as a conference paper at ICLR 2023
mentations that would work well for general as well as portrait harmonization. We also introduce new real-world testing data Portrait with ground-truth annotations collected from professional experts to benchmark the methods. Codes, datasets and pretrained models will be made public after the review process.
In summary, the following the key contributions of this work:
We are the first to introduce Interactive Harmonization and propose new framework which has the flexibility to solve it.
We propose new luminance matching loss better match the foreground appearance to the reference region.
We curate a new synthetic training dataset as well as a professionally annotated benchmark test for portrait harmonization and demonstrate state-of-the-an results on it.
We show that performing interactive harmonization using our proposed method results in visually pleasing portrait edits.
2 RELATED WORKS
Classical methods for image harmonization perform color and tone matching by matching global statistics Reinhard et al. 2001); Ling et al. (202T). Methods like Péreez et al. (2003): Tao et al. (2010) try to apply gradient domain methods to transfer the color and tone /hhu et al. (2015) proposed the first convolutional network-based method to improve realism of composite images. The first end-to-end deep learning-basee image harmonization method was proposed by Tsai et al.
(017) where encoder-decoder based convolutional network was used. Following that Cun & Pun (2020) proposed an iatention-based module to improve harmonization. Dove-Net, proposed by Cong et al. (2020) used Generative Adversarial Network (GAN)-based method with ad- ditional domain verification discriminator to verify if the foreground and background of given image belongs the same image. [ong al.] (2020) also proposed a public dataset, iHarmony4 for image harmonization. Following that, [ong et al. (2021) proposed Bargain- which uses domain extractor extract information about the background to make the harmonized foreground more consistent with the background. [Sofiuuk et al. (2021) proposed a new architecture involving pre-trained classification models to improve harmonization. Attention- based feature modulation for harmonization proposed in Hao et al. (2020). [ing et al. (2021) proposed RainNet which intro- duces a region-aware adaptive instance normalization (RAIN) module to make the style between the background and the composite foreground consistent. Guo et al. (2021b) proposed intrinsic image harmonization where reflectance and illumination are disentangled and harmonized separately. can be noted that photo-realistic style transfer methods Li et al. (2018); Luan et al. (2017); Yang eet al. 2019) when adopted for harmonization do not perform well as they require similar layout as explained in Diang al. (202T). Jiang et al. (202T) proposed self-supervised framework solve image harmonization. Recently, Guo et al. (202 proposed an image harmonization framework using transformers. Portrait relighting methods Zhou et al. (2019); Pandey al. (2021) have also been explored to produce realistic composite images for a desired scene.
3 INTERACTIVE HARMONIZATION
Setting: Given an input composite image C, the goal is generate harmonized image H that looks realistic. Composite image corresponds to the edited image where new person is introduced into the scene. This newly introduced region is termed as composite foreground region F and the scene it was introduced into is termed as background B. In general harmonization, we use the background B to harmonize F. interactive harmonization, we propose using a specific region of the background B guide harmonizing F. Note that the region R is subset of B i.e (R e B). R can be any region pertaining to the background that the user wants F to be consistent with.
For portrait images, easy way to perform realistic harmonization would be to select the per- son/object in the reference portrait as to guide the edited person/object F. This will make sure that that the luminance conditions of the reference portrait image is consistent with that of the newly edited-in portrait. However, we do not hardly constrain the region to only a portrait in the back- ground, it can also be objects even a part of the scene. Please note that portraits here do not only
3

---

Published as a conference paper at ICLR 2023
Compostic Ioput
Foreground Region
Harmenizer Nerwork
Harmonized Foreground
Composite with reforence image
welecte Reference region)
Solk Code
Reference Ragion
Style Encoder
Hamoonized Outpui
Figure Overview the proposed interactive portrait harmonization framework. correspond to images containing people, it can also contain objects. photo becomes a portrait when certain subjects are the main focus of the picture.
Dataset Curation: Publicly available datasets like iHarmony4 Cong et al. (2020) were proposed for background harmonization and do not provide any reference region information. So, curate a synthetic dataset and also introduce a real-world portrait harmonization dataset for validating.
1) IntHarmony: This newly curated synthetic dataset specifies an additional reference region to guide the harmonization. IntHarmony has the following information for each data instance: com- posite image, ground truth, foreground mask of the composite foreground and a guide mask which provides information about the reference region guide the harmonization. IntHarmony is built on top of MS-COCO dataset Lin et al. (2014). We make use of the instance masks provided in MS-COCO simulate foreground and reference regions. First, we select random instance mask to pick the foreground region. The selected foreground region is then augmented using a wide set of meaningful augmentations focusing on luminance, contrast and color. We use the augmentations proposed in Jiang et al.) (2021) to create the composite. More details on this can be found in the sup- plementary document. This augmented foreground is composited with the original image to get the composite image (7). Another random instance mask is used to get the reference guide mask. The original image is considered as the ground truth. The number of training images in IntHarmony is 118287 and 959 images are allocated for testing. Note that the instance masks and the augmentations are chosen at random to induce more generalizabilty to the network. IntHarmony consists of both objects, backgrounds and people and is not very focused for portraits. We use this dataset to help the network learn the interactive way to perform harmonization as we have a reference region to access. We also use similar dataset creation scheme on Multi-Human People Parsing (MHP) dataset Zhao et al. (2018) 8); et al. (2017): Nie et al. (2017) to synthesize another data further finetune the models portrait harmonization. Note that MHP dataset contains only images with people.
2) PortraifTest: To validate our performance for real-time applications, we introduce real-world test data consisting of high resolution composite portrait images. The ground truth annotated with the help of multiple professionals using PhotoShop curve adjustment layers to change brightness, hue, saturation layer and contrast of the composite foreground make it look more natural and pleasing. The number of images in this test dataset is 50 and the composite foreground is of varied luminance conditions, contrast and color.
4 METHOD
In this section, we first explain the details the framework we propose to solve interactive harmo- nization. We then give the training details and the loss functions introduce for this task.
Network Details: An overview of the proposed framework is shown in Figure It consists two sub-networks: harmonizer network and style encoder network.
Harmonization Network is an incoder-decoder architecture which takes two inputs: composite foreground image (I), foreground mask and style code (ω) extracted from the style encoder which of dimension x D. The encoder is built using 4 convolutional blocks where each conv block has conv layer that increases the number of channels and performs downsampling. followed by a set of
4

---

Published as a conference paper at ICLR 2023
residual blocks. The latent features from the encoder have a spatial dimension of (H/16) (W/16). The decoder is also built using 4 conv blocks where each block has an upsampling layer and a set of residual blocks similar the encoder. In addition,
adaptive instance norm (AdaIN) layers in each conv block that takes in the style code (d) and the features from previous conv block in decoder as their input. Thus, the decoder uses the style code extracted from the reference object guide the harmonization. The output from decoder is the harmonized foreground image. It is then composited with the original reference image to get the complete harmonized output.
Style Encoder takes in the reference region chosen by the user as the input. The style encoder consists of a series of partial convolutional layers Liu al. (2018) to downsample the input image, extract meaningful features and output a style code. The partial convolution layers take in both the reference image and guide mask as their output. The latent embedding is fed through a average pooling layer to obtain the 1D style code (d)).
Training Details: Training the network for interactive harmonization is performed stage-wiss wise. In the first stage, we train the network to perform background harmonization, where the training data of iHarmony4 Cong et al. (2020) is used, The input to the harmonizer network is the masked out foreground image concatenated with the style code of the background extracted from the style encoder. In the next stage, we finetune our network for interactive harmonization on IntHarmony. Here, the input to the style encoder is the reference region. Thus, the network here is trained in such a way that a reference region guides the harmonization which is also due to the loss functions we introduce (see Section 4.3). Finally, for portrait harmonization to be used in professional settings. The input to the harmonizer network the masked out foreground image (rather than entire composite image) concatenated with the style code of the background extracted from the style encoder. From our experiments, we realized that using the entire composite as input to the harmonizer network does not result in optimal training. This happens because the style code information bleeds out to the background of the composite, reducing its influence to the foreground. Masking out the background makes sure that the style code affects only the composite foreground and not the entire image.
further fine-tune the network on the augmented MHP dataset
Loss Functions: We use a combination of the proposed luminance matching loss, consistency loss, harmonization loss, and triplet losses for training our framework.
Luminance Matching loss: The main objective of interactive portrait harmonization to make sure that the harmonized foreground matches the appearance of the reference region. To achieve this, we introduce three new losses highlight, mid-tone and shadow losses, which are used to match the highlight, mid-tone and shadow between the reference region and foreground region to be harmonized. We define these losses as follows:
Lhighlight Haar Rmazli
D = H mean Rmman ll
(2)
Lahadow = Hiin Rminnli,
(3)
where H corresponds to the harmonized image and R corresponds to the reference region. These losses try to match the intensities of the foreground with the reference region at the highest, lowest and average luminescence points thus matching the contrast. For the highest and lowest points, we choose the 90th and thth percentile respectively to improve stability. We define luminance matching loss (Liar) as the sum of these 3 losses:
(4) Luminance matching loss (LLM) is illustrated in Figure Note that LLA is an main addition to our training methodology which helps to make sure that the statistics between the reference and fore- ground are matched. This loss penalizes the network if the statistics of the harmonized foreground and the reference region are not matched.
LLM = Lhightight + Lmid tone + Lshadow-
Consistency loss:
We also introduce consistency loss Lconsie between style codes of the harmo- nized foreground and the reference region to penalize the network whenever the style code of the reference region and harmonized foreground region are different from each other. The consistency loss is defined as:
Lconsis
[ló(h) ((b))11
(5)
We also use generic harmonization loss which is an Ll-loss between the prediction and the ground-truie truth as follows:
Harmonization loss:
Lnarmonisation ||
(6)
5

---

Published as a conference paper at ICLR 2023
Luminance Matching Loss
Highlight Matching
Midione Matching
Shadow Matching
Foregroind Region
gright
X Peyadoree
Refferance Region
Foregoound
Reference Region
Region
Hightagl
Shudow
Figure 3: Overview of the proposed Luminance Matching loss.
where corresponds to the harmonized image and represents the ground truth. This loss tries to minimize the distance between the prediction and input thus making sure the prediction always consistent in appearance with that of the ground truth.
Triplet losses: We introduce triplet losses similar to Cong et al. 2021 based the style codes of the composite foreground (c), harmonized foreground (h), real foreground (r) (ground truth) and
that the reference region (b). The first triplet loss is follows:
Ltriplet11 = max
max(Il@(h)) (h), ((b)]22-11 (l@(()) ((c)ll2+m, 0)-
This tries make the style code generated by the style encoder for harmonized foreground (h)
to be closer to that of style code of reference region (6) while being far away from the style code of composite foreground (c). m here corresponds to the margin.
We define another triplet loss which forces the real foreground style code (r) to be close to harmo- nized foreground style code (h) while making sure the style code harmonized foreground style code (h) is far away from composite foreground (c):
Ltriplet2 = max(|lo(h)) 6(r) (1)(h), (())ll2
The total loss used train the network is follows:
LIPH = Charmonization + aLLss + >.Connsi + {CCrápleet1 + Leriplet2)
where a, B and X are the factors that control the contribution the triplet losses have in the total loss.
EXPERIMENTS
Implementation Details: Our framework is developed in Pytorch Paszke et al. (2019) and the training is done using NVIDIA RTX 8000 GPUs. The training is done stage-wise explained the previous section. We use an Adam optimizer Kinigma
Ba (2014) with learning rate of 10 10-4, 10 10 at each stage respectively. The batch size is equal to 48. The images resized to 256 x 256 while training. During inference, to get the high resolution images back, we polynomial fitting like in Afii al. (209)). We composite the high resolution harmonized foreground with the reference image to get the high resolution harmonized image.
Comparison with State-of-the-art: We compare our proposed framework with recent harmoniza- tion networks like Dove-Net(Come et al. (2020). Bargain-Net Cong et al.) (202I). and Rain-Net Ling al. (02T)) We note that Bargain-Net has domain extractor branch that directly takes in the background of the image to make the harmonization consistent with the background. In addition to comparing with the original model. also re-train Bargain- -Net in a interactive setup similar to our proposed approach for fair comparison. Here, we feed in the reference region as the input to the do- main extractor and use similar protocols as our method. We call this configuration Bargain-Net (R). Note that Bargain-Net (R) is trained with the same data and stage-wise training our method for fair comparison. We explain why the other frameworks cannot be converted for interactive harmo- nization in the supplementary material. We use publicly available weights of Dove-Net, Bargain-Net and Rain-Net to get their predictions on PertraitTest and IntHarmony. We denote our approach as IPH (Interactive Portrait Harmonization). While testing with IntHarmony, Bargain-Ne (R) and IPH are trained on the training data of IntHarmony. While testing with PortraitTess, we compare with two more configurations: Bargain-Net (R) (++) and IPH (++) which are further finetuned augmented

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative comparison in terms of reference based performance metrics with previous
methods. We compare our proposed method IPH in terms PSNR, SSIM and CIE Delta with previous methods across test datasets. Red and Blue corresponds to first and second best results. ++
corresponds to the configuration where the model is further finetumed on augmented MHP dataset.
Dataset
Type
Method
Venue
PSNR (1)
SSIM
CTF Della (U)
Direct Composite
27.21
0.9709
37.27
Generic
DoveNet Cong et al.
2020
CVPR 20
27.44
0,9314
23.49
Hamonniztion
Bargain
202
ICME 21
28.47
0,9364
20.95
Portrai(Test
RainNet Ling el aL 2021
CVPR 21
28.55
0.9315
18.32
BarguinNet Cong ci (2021)
ICME 21
28.56
0,9389
19.91
Interactive
BargainNet (R) (+7))Cogg Cong 2021
ICME 21
30,10
0.9787
12.31
Harmonization
IPH (Ours)
30.86
0.9553
12.16
(++) (Ours)
36.52
0,9871
4.21
Direct Composite
25.22
0,8957
54,02
Generic
GooeNerCCong et al. 2020
CVPR 20
26.60
0.9011
32.45
IntHarmony
Harmonization
BargainNell el al 20Z
RainNet Ling el 120Z
CVPR 21
ICME
27.94
24.92
0.9113
0,9102
28.65
20.48
Interactive
BargninNet Cong el al. IPH
20211
ICME
27.19
0,9078
21.52
Harmonization
30.22
0.9190
15.62
portrait images from the MHP dataset as explained in Section 3.3. As IntHarmony actually contains objects, people and natural scenes, validating it helps us prove that the interactive feature helps in even normal harmonization. Validating on PortraitTest helps us prove if our method will work well on real-world high resolution portrait images. It also helps understand where our method stands when compared to professional editors.
We use referenced quality metrics like PSNR, SSIM and CIE Delta to quantitatively compare the performance of IPH with the previous methods. Table consists of the performance comparison on both the PortraitTest and IntHarmony test datasets. We note that IPH achieves the best performance in terms of PSNR, CIE Delta and SSIM when compared to the previous harmonization networks. IPH outperforms all previous methods across all metries in both datasets. It can be noted that BargainNet (R)/ BargainNet (R) (++) do not perform as well as IPH/IPH (++) even though they are trained with the same pipeline and training conditions of IPH. Prior methods fail as they do not consider arbitrary regions to guide the harmonization and also don't account for luminance matching. We get the best performance as the proposed approach is more effective at improving the performance compared the previous frameworks.
Referenced Quality Metries:
Visual Quality Comparison: In Figure 4 we show results on composite portraits from Portrait dataset. It can be seen that the harmonized foreground generated by generic harmonization meth- ods like Dove-Net, Bargain-Net, and Rain-Net is inconsistent with color and luminance conditions when compared to the reference portrait. Also, the predictions of Bargain-Net and Rain-Net fail to harmonize the local lighting conditions present in the composite as they fail to explicitly match the highlight, mid-tone and shadow. Bargain-Net (R) (++) performs interactive harmonization however it still fails to match the contrast of the reference. Our method harmonizes the composite well even considering the local luminance conditions and is close the annotations done by a professional.
We conduct two different human subjective reviews to compare the performance of IPH with other methods. First, we select 15 images from the PortraitTest dataset and present the predictions of Dove-Net, Bargain-Net (R) (++), Rain-Net and IPH (++) on screen. We ask 30 subjects to independently rank the predictions from to 3 based on the visual quality considering the color and luminance conditions of all the portraits in the image look consistent. We give scores 3,2, and for ranks 1,2 and respectively. Additionally, we also conduct a pair-wise A/B user study. Here, we show the input image and the output of two methods to user. We then collect these votes and report the preference of the users. We apply the Thurstone Case V analysis to these votes to get the preference values. We also report the Bradley-Terry (BT) score All these scores have been tabulated in Table 2 where our proposed method is observed to be desired the most.
User Studies:
DISCUSSION
Ablation Study: We conduct ablation study to analyse the importance of each component of our proposed method. We evaluate these models on the PortraifTest dataset. We start with just the Harmonizer Network (H-Net) without AdaIN layers trained for generic harmonization. Then we add
7

---

Published as a conference paper at ICLR 2023
T
Direct Composite
Mask
Dove-Net
BG-Net
RainNet
BG-Net (R) (++)
IPH (++)
Ground Truth
Direct Composite
Mask
Dove-Net
BG-Net
RainNet
BG-Net (R) (++)
IPH (++)
Ground Truth
Qualitative comparison on the Portrait Test dataset. Ground Truth annotations here created by human professional experts. Red mask corresponds to the composite foreground region and blue mask corresponds the reference region chosen from the background.
Figure 4:
8

---

Published as a conference paper at ICLR 2023
Table 2: User Study: Quantitative Comparison with respect scores from human subjects while
evaluating on PortraitTest dataset.
Method
Ranking Score
Preference (?)
BT
Composite
-0.439
0.1577
DoveNet
19,40
-0.224
0.1602
Bargain-Net (R) (++)
26.40
0.102
0.1965
Rain-Net
12.26
-0.215
0.1600
IPH (++) (Ours)
28.33
0.782
0.3252
Table Ablation study: We perform an ablation study on real test data to understand the contribu- tions brought about by different techniques proposed in IPH.
Method
PSNR (1)
SSIM (1)
MSE (L)
Direct Composite
27.21
0.9709
155.75
H-Net
28.40
0.9342
120.35
SE
30.91
0.9656
53.89
W Mid-tone loss
31.25
0.9715
40.54
W Shadow loss
32.56
0.9668
50.34
Highlight loss
34.21
0.9612
45.28
w SE LM loss (IPH)
36.52
0.9872
18.33
the style encoder and AdaIN layers in the decoder and train for interactive harmonization (H-Net+ SE). All these methods are trained using the losses Charmonization: Cemsis and Ltriplet- Then add the novel losses introduced in this work Lhighlight Lmid-tones and Lshadow individually. From Table it can be observed that all the new components introduced in this work are important and play a key role in achieving better performance. Making the framework interactive by using the style encoder and injecting the style code of reference region to guide the harmonization is sufficient the push the performance above previous methods. Furthermore, the proposed luminance matching loss helps improve the performance by a significant margin proving that matching the highlight, mid-tone and shadow of the foreground and reference help obtain better harmonization.
One key advantage our framework has over generic har- monization is that it is more flexible and robust. We show that IPH is robust different regions chosen by the user. We note that the output harmonization adjusts itself with the select region cho- sen from the background. This is very useful cases where the luminance conditions vary across different parts of the background image. We illustrate this in Figure 10
Robustness to chosen reference region:
Compasite
Harmonized Image
Harmonized Image
Harmonized Image
Figure 5: Qualitative results with different regions of background chosen as reference. The first image is the direct composite and the red box in other images show the reference region chosen by the user and the resultant harmonization of the boat can be seen in the respective images.
7 CONCLUSION
In this work, we proposed a new framework called interactive portrait harmonization where the user has the flexibility of choosing a specific reference region to guide the harmonization. We showed that this helps the user obtain more realistic harmonization results while providing more control and flexibility to the compositing workflow. We also proposed new luminance matching loss to carefully match the appearance between the foreground composite and reference region. In addition, we introduced two datasets: a synthetic IntHarmony dataset for training and a real-world Portrai dataset for testing. Extensive experiments and analysis showed that IPH performs better and is more robust and useful to solve real-world portrait harmonization problems in practical settings compared to the previous methods.

---

Published as a conference paper at ICLR 2023
REFERENCES
Mahmoud Afifi, Brian Price. Scott Cohen, and Michael S Brown. When color constancy goes wrong: Correcting improperly white-balanced images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3535-1544, 2019.
Wenyan Cong, Jianfu Zhang. Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang. Dovenet: Deep image harmonization via domain verification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8394-8403, 2020.
Wenyan Cong, Li Niu, Jianfu Zhang. Jing Liang, and Liqing Zhang. Bargainnet: Background- guided domain translation for image harmonization. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEFF, 2021.
Xiaodong Cun and Chi-Man Pun. Improving the harmony of the composite image by spatial- separated attention module. IEEE Transactions on Image Processing. 29:4759-4771 4771,, 2020.
Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, and Junyu Dong. Image harmonization with transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14870-14879, 2021a.
Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, and Bing Zheng. Intrinsic image harmo- nization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 16367-16376, 2021b.
Guoqing Hao, Satoshi lizuka, and Kazuhiro Fukui. Image harmonization with attention-based deep feature modulation. In BMVC, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, and Zhangyang Wang. Ssh: A self-supervised framework for image harmonization. arXiv preprint arXiv:2108. 06805, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1rXiv 2014.
Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng. Multiple-human parsing in the wild. arXiv preprint arXiv: 2017.
Yijun Li, Ming- Yu Liu, Xueting Li, Ming-Hsuan Yang, and Jan Kautz. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 453-468, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Jun Ling, Han Xue, Li Song, Rong Xie, and Xiao Gu. Region-aware adaptive instance normalization for image harmonization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9361-9370 2021.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chur Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 85-100, 2018
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Pro- ceedings of the IEFE conference on computer vision and pattern recognition, pp. 4990- 2017.
Xuecheng Nie, Jiashi Feng, Junliang Xing, and Shuicheng Yan. Generative partition networks for multi-person pose estimation. arXiv preprint arXiv: 1705.0722,, 2017.
10

---

Published as a conference paper at ICLR 2023
Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: learning to relight portraits for background replacement. ACM Transactions Graphics (TOG), 40(4):1-21, 2021.
Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Multi-scale interactive network for salient object detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9413-9422 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performanee d'Alcéé-Buc,
deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp- Curran Associates, Inc., 2019. URL http: /ttp:://appp.s.ee neur pewr. wtw.eeuwwp..cccpppeeeeeoooeeoeee ccpppppp
901 5 was---yttccha---tteeeeee htt.://www...............eeeeeee learniinnn ibbarr.
pdf
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing, In ACM SIGGRAPH 2003 Papers, pp. 313-318. 2003.
Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer graphics and applications, 21(5):34-41 2001.
Konstantin Sofiiuk, Polina Popenova, and Anton Konushin. Foreground-aware semantic representa- tions for image harmonization. In Proceedings of the IEEE/CVF Winter Conference on Applica- fions of Computer Vision, pp. 1620-1629, 2021.
Michael W Tao, Micah K Johnson, and Sylvain Paris. Error-tolerant image compositing. In Euro- pean Conference on Computer Vision, pp. 31-44. Springer, 2010.
Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang. Deep image harmonization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2017.
Shuai Yang. Zhangyang Wang, Zhaowen Wang, Ning Xu, Jiaying Liu, and Zongming Guo. Con- trollable artistic text style transfer via shape-matching gan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4442-4451, 2019.
Jian Zhao, Jianshu Li, Yu Cheng, Terence Sim, Shuicheng Yan. and Jiashi Feng. Understanding humans in crowded scenes: Deep nested adversarial learning and a new benchmark for multi- human parsing. In Proceedings of the 26th ACM international conference on Multimedia, pp- 792-800, 2018.
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait lighting. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 2019.
Jun Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, and Alexei Efros. Learning discriminative model for the perception of realism in composite images. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 3943-3951, 2015.
A APPENDIX
Converting user-guided to automatic? Please note that although we provide the flexibility for the user to point to a reference region to guide the harmonization, our framework can be made completely automatic. We can make use of saliency detection models Pang et al. (2020) to get the segmentation mask of the most salient object/person region to guide the portrait harmonization. In fact, harmonization and get feedback from the user if they are interested to select a different region or a sub-part of the current region selected guide the harmonization.
the background and use that as the reference
make this the initial outcome for portrait

---

Published as a conference paper at ICLR 2023
Different Objects Harmonization: In IPH, the input the style encoder does not have any straint. need not even be a region present in the background image. Any region from any image can be fed as the imput to the style encoder thus making the framework more flexible. This can be useful the context of style transfer and re-luminance applications. Figure illustraate the harmo- nization result when the composite foreground is harmonized with totally different objects from the different reference images.
Reference Region from a different image:
Direct Composite
Harmonized
Harmonized Image
Figure 6: Qualitative results with reference regions taken from different images. We show that with IPH we can harmonize the foreground with reference regions present in any image.
Number of Parameters: terms of parameters, IPH is lighter compared to the previous methods. IPH has 27 parameters compared to Bargain-Net's 58M, Dove-Net's 54M and Rain-Net's 54M number parameters.
Tweaking from Harmonization to Color Transfer:
We show that the appearance of the harmo- nized image can be easily controlled using the style code. the style code is made to learn the match color instead of harmonizing the image,
show that our framework can be used to perform color transfer. We make two simple changes in our framework to obtain this: (1) increase color augmentations (2) change Lconsis loss to make the style codes of the harmonized foreground and the reference region to be far apart from each other; we could obtain style encoder which impart an aggressive color change. We term the style code extracted from this style encoder as & and the original style code as ø. Now to control the appearance, we just blend these color codes extracted from these two encoders using different rations. The new style code would
7 rl (1 r2) (2)* Φ,,
where r1 and r2 are the blending ratios. In Figure 7 we show the change in appearance for different
combinations of r1 and r2. This shows that IPH can be easily tweaked to make interactive resulting
in realistic harmonization depending on the user's choice of and r2.
Direct Compusite
100,
r2
80,
50,
Foreground Region
Refecence Region
42,
60
20,
80
r 100
Figure 7: Modifying our framework for color transfer. Qualitative results with different values of rl and r2 show how we can perform interactive color transfer.
Although interactive portrait harmonization outperforms previous harmonization methods, there are still some limitations. Our method works fairly well even for in-the-wild com- posite portraits consisting of people. However, its performance decreases if we test on portraits consisting of objects. This can understood as it is still difficult for the style encoder to extract meaningful style code if the object material is peculiar. For example, if the reference object chosen
Limitations:
12

---

Published as a conference paper at ICLR 2023
has shiny surface is of unique material, is difficult for the style encoder to extract meaning- ful style code capturing the texture/material reflectance the object and use for harmonization.
Direct Composite
Mask
Harmonized Image
Figure 8: Limitations: Learning intrinsic material details is difficult in cases where objects are shiny
peculiar.
Bargain-Net and Rain-Net: We note that Bargain-Net Cong et al. (202T) and Rain-Net Ling et al. (2021) specifically focus using the background guide the harmonization. Bargain-Ne -et has domain extractor to extract the features of background while Rain-Net proposes a RAIN module which normalizes the background features with the foreground. We tried make both of these networks interactive by replacing the background mask with a guide mask to select the reference region. With the pre-trained weights, while changing the guide mask we did not see any significant change in terms of performance metrics nor visual quality for both Bargain-Nee and Rain-Net. If we retrain the network to take in the guide mask instead of background mask, we noted small improvement in results for Bargain-Net as reported in the main paper. For Rain-Net, we did not see any improvement. This might be because RAIN module uses the masks in the feature space to normalize and that might not be an optimal way to actually get meaningful features to inject to the foreground. Training
an interactive way in fact is not robust at all and we get a small decrease in performance. We also not that for these setups changing the guide mask to background mask does not bring out much effect the harmonized output which makes clear that these frameworks cannot be converted serve interactive harmonization efficiently.
Augmentations:
To generate the composite image in the synthetic dataset IntHarmony, we use a set of augmentations. For each image, these augmentations chosen at random. To account for appearance changes, we use a brightness and contrast augmentation. We also use color jitter to enforce hue augmentation. In addition, also use 3D LUT augmentations as proposed in Jiang et al. (202T) We also generate local masks and enforce augmentations to cover local lighting effects. Here, we apply augmentations such as soft lighting, dodge, grain merging and grain extraction. Note that all the above set of augmentations are considered as set and for each image random augmentation is selected and applied on composite foreground.
also
gamma transformation. Apart from these,
Potential Negative Societal Impacts: The proposed method may have negative social impacts people are not using this model properly. As this work focuses on making an edited image look realistic, can be used to create realistie fake images. Realistic fake images should be contained as it can act tool for harassment and fraud. One potential solution is gated release of trained models.
More Results: In Figure 9 we show results composite portraits from IntHarmony dataset. It can be observed that our method produces realistic harmonization results closer to ground truth when compared previous methods. The quality of predictions proves the usefulness of proposed approach to solve the interactive harmonization problem. Please refer to the supplementary material to see many more visualizations.
in-the-Wild Portrait Harmonization Results: To further validate our method for in-the-wild photos, we randomly create composite images and check the portrait harmonization results. We visualize these composites and the corresponding results in Figure 5. It can be observed that our method produces realistic harmonization results with the appearance being close how would have been if the person was photographed in-situ.
13

---

Published as a conference paper at ICLR 2023
Direct Composite
Mask
Dove-Net
BG-Net
RainNet
BG-Net (R)
IPH
Ground Truth
Figure 9: Qualitative comparison on the IntHarmony dataset. Red mask corresponds to the composite foreground region and blue mask corresponds to the reference region chosen from the background. The other columns correspond the predictions generated by the corresponding meth- ods. Best viewed zoomed in and in color.
14

---

Published as a conference paper at ICLR 2023
Composite Image
Prodiction
Figure 10: Qualitative results while testing our method for in-the-wild composite portraits. The red box denotes the composite foreground. We use the portrait in background reference region.
15


Reviews:
[{"review_id": "zaXKMbB9d6B", "reviewer": "Reviewer_MxTM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new framework, namely interactive portrait harmonization, to allow users to select a reference region in the background image to guide the harmonization. The key to this framework is a luminance matching loss to ensure the luminance consistency between the selected reference region and the foreground region. Two datasets are also constructed for training and testing, respectively. Extensive experiments on natural and synthetic data demonstrate the effectiveness of the proposed framework.\n", "summary_of_the_review": "The paper introduced a new setting, interactive portrait harmonization, to allow users to select a reference region in the background image to guide the harmonization. However, the luminance/color conditions of the reference region do not necessarily match that of the foreground region, which limits the realism of interactive harmonization. In addition, the reference guide region is subjective and not necessarily optimal. Thus, I give the paper a borderline reject rating.", "strength_and_weaknesses": "Strength\n+ A novel framework with a luminance matching loss is proposed to solve the interactive portrait harmonization.\n+ The paper is well-written and easy to follow.\n\nWeaknesses\n- The harmonized foreground is only suitable for the reference region, not the foreground placement region. In fact, in the real world, the illumination at different locations in the background might be different, which results in appearance differences of the foreground object. Besides, is it reasonable to RANDOMLY choose an instance mask for dataset construction as a reference guide mask? How to ensure the luminance/color consistency between the foreground placement location and the random reference location?\n- The performance of the interactive image harmonization heavily depends on the user’s selected reference guide region. Compared with those automatic image harmonization methods, the reference guide region selected by the user is not necessarily optimal. In addition, in some cases, the partial region of the background is inadequate to infer the luminance/color conditions of the foreground placement region.\n- For the luminance matching loss, the author claimed three new losses are used to match the highlight, mid-tone, and shadow between the reference region and foreground region. But these equations (1)-(3) indicate that the losses only relate to the predicted harmonized image and the ground truth, which is unrelated to the reference region.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q7wM_6A360b", "reviewer": "Reviewer_DBoQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces interactive image harmonization that enables the user to select a specific object/person to guide. The paper also proposes a new technique of using luminance matching loss and a new dataset for validating portrait harmonization. Experimental results show the effectiveness over the previous methods.", "summary_of_the_review": "The paper is introducing interactive image harmonization that allows the user to select a region as a guide. When two or more people/objects are in the same image, if the user uses a person/object as a reference image, the generated image would be better especially for portrait photos taken in a studio with a solid background. Experimental results show the effectiveness of the proposed method.", "strength_and_weaknesses": "Strength\n\n1. Experimental results show the effectiveness significantly.\n2. The ablation study shows the effectiveness of using the proposed luminance matching loss.\n\nWeakness\n1. In term of deep learning model, it looks like very similar to the previous model of DoveNet. The paper does not discuss the difference in details.\n\n2. If two objects are in different colors, such as red car as a reference region and blue car as a foreground image, what would happen?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8ML7CpjYf2D", "reviewer": "Reviewer_UNoj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework for interactive harmonization. A new network is proposed to allow users to pick certain regions of the background and perform harmonization on the foreground. A novel luminance matching loss and a new synthetic dataset and a real test dataset is proposed. The proposed network is shown to perform better than previous generic harmonization methods on the new datasets as well as on some “in the wild” edited portrait images.", "summary_of_the_review": "Based on the strength and weakness mentioned above, I recommend to accept this paper. ", "strength_and_weaknesses": "-> The  idea of interactive harmonization is interesting and could meet the practical demands of concise human harmonization in the real world. This is the first work to look into this problem.\n\t\n-> One of the main contributions is a large-scale dataset with reference regions annotated. And they also build a benchmark test with professionally edited harmonization results. This dataset could be helpful if it is being released publicly. The paper has relatively extensive evaluations on the dataset and also vivid examples.\n\n-> The loss function introduced seems to help well in achieving a good performance while looking at the ablation study. The inspiration of the new loss function being what is done in manual harmonization is sound.\n\n->The paper shows results on some “in the wild” edited images which look impressive. \n\n-> The framework is shown to be robust and have different results for different reference regions and also shown to have multiple other use-cases like color transfer and easy switching to automatic harmonization.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "09agUWbHyd", "reviewer": "Reviewer_53Gf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new image harmonization method. Image harmonization aims at compositing regions of different images in a way the observer can not tell they were coming from different original images. The main novelty of this work is the allowance of interaction by the user. In the proposed model the user can select which part of the background image should be used for harmonization. Furthermore, thae authors also propose a new loss based on luminance inspired by professional users.", "summary_of_the_review": "This is an interesting paper with good results. Some issues need to be addresses (see my detailed explanation in the Weaknesses section). For this reason, my rating is somehow between 5 and 6. ", "strength_and_weaknesses": "Strenghts:\n\n- The paper is easy to follow.\n- The results overtake the state-of-the-art.\n- Allowing the user to select the region of interest is interesting.\n\nWeaknesses:\n- Luminance Matching Loss: The Luminance matching loss is not perceptual. It is well known that to be able to perceive a change of luminace value, we need a much larger difference in the highlight regions that in the shadow regions (Weber's law). For this reason, I stringly believe than the three different terms in the luminace loss shall be weighted according to this fact.\n\n- The authors use a ranking experiment for the user study. These type of experiments are known to be very noisy. Also, there is not a reference in how statistical significant is the distance of 0.19 between IPH and Bargain-Net. For this reason, the authors should instead go for a pair-wise comparison, and then use the Thurstone Case V law (See [1]) to also study the statistical significance of the results.\n\n- Authors need to discuss why when including the Shadow loss and the Highlight loss the method improves PSNR but get worse in SSIM. I believe it might be related to my first comment.\n\n- Also, it is not make sense to put both PSNR and MSE in the Tables. They are completely correlated as PSNR is directly computed from MSE. Also, authors need to include some Color metric, in order to prove that no color artifacts, such as Hue shifts appear in the result. I recommend CIE DElta E.\n\n[1]  Color Gamut Mapping, Ján Morovič, 2008, John Wiley & Sons, Ltd\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster